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Today, computer imaging covers various aspects of data filtering, pattern 
recognition, feature extraction, computer aided design, and computer aided 
inspection and diagnosis. 

Pillars of the field of computer imaging are advanced, stable, and reliable algo- 
rithms. In addition, feasibility analysis is required to evaluate practical relevance 
of the methods. To put these pillars on solid grounds, a significant amount of 
mathematical tools are required. This handbook makes a humble attempt to provide 
a survey of such mathematical tools. 

We had the vision that this imaging handbook should contain individual chapters 
which can serve as toolboxes, which, when aligned, form background material for 
complete applied imaging problems. Therefore it should also give an impression on 
the broad mathematical knowledge required to solve industrial and applied research 
applications: The image formation process, very frequently, is assigned to the 
inverse problems community, which is prominently represented in this handbook. 
The subsequent step is image analysis. Nowadays, advanced Image Analysis, 
and Image Processing in general, uses sophisticated methods from Geometry, 
Differential Geometry, Convex Analysis, Numerical Analysis, to mention just a few. 
In fact, by the rapid advance of Imaging, the mathematical areas have been pushed 
forward heavily, and raised their impact in application sciences. 
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this handbook’s content. For a project of this size, it is likely that essential topics 
are missed. In a rapidly evolving area like Imaging it is likely that new areas will 
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appear in a very short time and should be added to this handbook, as well as recent 
development enforce modifications of existing contributions. We are committed to 
issuing periodic updates and we look forward to the feedback from the community. 
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Abstract 


This introductory treatment of linear inverse problems is aimed at students and 
neophytes. A historical survey of inverse problems and some examples of model 
inverse problems related to imaging are discussed to furnish context and texture 
to the mathematical theory that follows. The development takes place within 
the sphere of the theory of compact linear operators on Hilbert space, and the 
singular value decomposition plays an essential role. The primary concern is 
regularization theory: the construction of convergent well-posed approximations 
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to ill-posed problems. For the most part, the discussion is limited to the familiar 
regularization method devised by Tikhonov and Phillips. 


1 Introduction 


p> ... although nature begins with the cause and ends with the 
experience we must follow the opposite course, namely 
... begin with the experience and by means of it 
end with the cause. 
Leonardo da Vinci 


An inverse problem is the flip side of some direct problem. Direct problems 
treat the transformation of known causes into effects that are determined by some 
specified model of a natural process. They tend to be future directed and outward 
looking and are often concerned with forecasting or with determining external 
effects of internal causes. Direct problems have solutions (causes have effects), and 
the process of transforming causes into effects is a mathematical function: a given 
cause determines, via the model, a unique effect. In direct problems the operator 
that maps causes into effects is typically continuous in natural metrics: close causes 
have close effects. These features of direct problems make them well posed. 

The idea of a well-posed problem has its origins in Jacques Hadamard’s short 
paper [37] published in 1902. Hadamard held the opinion that an important physical 
problem must have three attributes: 


1. (Existence) It has a solution. 
2. (Uniqueness) It has only one solution. 
3. (Stability) The solution depends continuously on the data of the problem. 


A problem satisfying these three conditions is called well posed. In his 1902 paper, 
Hadamard called a problem bien posé if it has properties (1) and (2). Again in his 
1923 lectures [38], he called a problem “correctly set” if it satisfies (1) and (2). 
Condition (3) was not named as a specific requirement of a well-posed problem, but 
his explicit notice of the lack of continuous dependence on boundary data of the 
solution of Cauchy’s problem for Laplace’s equation led to (3) becoming part of the 
accepted definition of a well-posed problem. 

A problem is ill posed if it lacks these qualities. Hadamard’s suggestion that 
ill-posed problems are devoid of physical significance (déprourvu de signification 
physique) was unfortunate, as almost all inverse problems in the physical and 
biological sciences are ill posed. To be fair, it should be noted that Hadamard was 
speaking about a specific problem, the Cauchy problem for Laplace’s equation in a 
strip. On the other hand, Courant [15] insisted more generally that “a mathematical 
problem cannot be considered as realistically corresponding to physical phenomena 
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unless ...” it satisfies condition (3). The problems of existence and uniqueness in 
inverse problems can often be ameliorated by generalizing the notion of solution 
and constraining the generalized solution, but the key attribute of stability often 
is a feature that is inherently absent in inverse problems. This essential lack of 
stability usually has dire consequences when numerical methods, using measured 
or uncertain data, are applied to inverse problems. 

Inverse problems are as old as science itself. In fact, a reasonable working 
definition of science is the explanation of natural phenomena by the construction 
of conceptual models for interpreting imperfect observational representations of 
“true” natural objects or processes. This definition encompasses the three essential 
ingredients of mathematical inverse problems: a “true” solution, a model, or 
operator that transforms this true solution into an imperfect representation that 
is amenable to observations or measurements. One could say that inverse theory 
embraces an operating principle that is essentially Platonic: true natural objects 
exist, but it is only through models and imperfectly perceived images that we 
experience them. The challenge is to “invert” the model to recover a useful estimate 
of the true object from the observed image. In this sense, all of inverse theory deals 
with “imaging.” 

A mathematical framework for the study of inverse problems must provide suffi- 
cient scope for each of the three elements: true solutions, model, and observations. 
In this chapter the solution space and the space of observations are both taken 
to be Hilbert spaces, but not necessarily the same Hilbert space, as one naturally 
desires more of the solution than one demands from the observations. The model is 
a transformation or operator that carries a possible solution to an observed effect. 
We consider only linear inverse problems, so our models are linear operators. 

Any practical model suppresses some information. If a model represents every 
bit of information in the objects themselves (i.e., the model operator is the identity 
operator), then nothing is gained in conceptual economy. In this case one is in the 
absurd position of Mein Herr in Lewis Carroll’s Sylvie and Bruno Concluded: 


“We actually made a map of the country, on a scale of a mile to the mile!” ...“It has never 
been spread out yet,” said Mein Herr: “the farmers objected; they said it would cover the 
whole country, and shut out the sunlight! So now we use the country itself, as its own map, 
and I assure you it does nearly as well.” 


Finite linear models lead to linear algebra problems. Idealized limiting versions 
of finite models typically lead to compact linear operators, that is, limits of finite 
rank operators. A compact operator may have a nontrivial null-space, a non-closed 
range, or an unbounded (generalized) inverse. Therefore, these operators, which 
occur widely in models of linear inverse problems, lack all the virtues of well 
posedness. In this chapter, we provide a somewhat slanted survey of linear inverse 
problems, mainly involving compact operators, with special attention to concepts 
underlying methods for constructing stable approximate solutions. 

Before draping these ideas on a mathematical framework, we discuss a half 
dozen examples of model inverse problems that have played significant roles in 
the development of the physical sciences. 
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2 Background 


> Our science is from the watching of shadows; 
Ezra Pound 


This brief and incomplete historical survey of physical inverse problems is meant 
to give some perspective on certain inverse problems closely related to imaging 
in the broad sense. Our viewpoint involves both the very large scale, treating 
inverse problems loosely associated with assessing the cosmos, and the human scale, 
dealing with evaluation of the inaccessible interior of bodies (human or otherwise). 

Inverse theory, as a distinct field of inquiry, is a relatively recent development; 
however, inverse problems are as old as science itself. A desire to know causes 
of perceived effects is ingrained in the human intellectual makeup. The earliest 
attempts at explanations, as, for example, in the creation myths of various cultures, 
were supernatural — grounded in mysticism and mythology. When humankind 
embarked on a program of rationalization of natural phenomena, inverse problems 
emerged naturally and inevitably. An early example is Plato’s allegory of the cave 
(ca. 375 B.C.). In the seventh book of his Republic, Plato describes the situation. 
A group of people have been imprisoned since their birth in a cave where they 
are chained in such a manner that allows them to view only a wall at the back 
of the cave. Outside the cave life goes on, illuminated by a fire blazing in the 
distance. The captives in the cave must reconstruct this external reality on the basis 
of shadows cast on the rear wall of the cave. This is the classic inverse imaging 
problem: real objects are perceived only as two-dimensional images in the form of 
shadows on the cave wall. This annihilation of a dimension immediately implies that 
the reconstruction problem has multiple solutions and that solutions are unstable in 
that highly disparate objects may have virtually identical images. 

Aristotle adapted his teacher Plato’s story of the cave to address a scientific 
inverse problem: the shape of the earth. This shape could not be directly assessed 
in Aristotle’s time, so he suggested an indirect approach (see Book II of On the 
Heavens.): 

As it is, the shapes which the moon itself each month shows are of every kind — straight, 

gibbous, and concave — but in eclipses the outline is always curved; and since it is the 


interposition of the earth that makes the eclipse, the form of the line will be caused by the 
form of the earth’s surface, which is therefore spherical. 


Aristotle’s reasoning provided an indirect argument for the sphericity of the earth 
based on the shapes of shadows cast on the moon. 

Inverse imaging has been a technical challenge for centuries. The difficulties that 
early investigators encountered were vividly captured by Albrecht Diirer’s woodcut 
Man Drawing a Lute (1525). We can see the doubts and angst brought on by the 
inverse imaging problem etched on the face of the crouching technician (Fig. 1). 

The character on the left, standing bolt upright in a confident attitude, has the 
comparatively easy direct problem. He has complete knowledge of the object, and he 
knows exactly how the projection model will produce the image. On the other hand, 
the crouching man on the right, with the furrowed brow, faces the more difficult 
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Fig. 1 A renaissance inverse problem 


inverse problem of assessing whether the image captures the essential features of the 
object. Diirer’s woodcut is a striking representation of the comparative difficulties 
of direct and inverse assessment. 

Modern imaging science has its roots in Galileo Galilei’s lunar observations 
carried out during the winter of 1609. Prior to Galileo’s study, the moon was 
thought to belong to the realm of the Pythagorean fifth essence, consisting of 
perfectly uniform material in perfect spherical form. Galileo’s eyes, empowered 
by his improved telescope, the earliest scientific imaging device, showed him 
otherwise [21]: 


... we certainly see the surface of the Moon to be not smooth, even, and perfectly spherical, 
as the great crowd of philosophers has believed about this and other heavenly bodies, but, 
on the contrary, to be uneven, rough, and crowded with depressions and bulges. And it is 
like the Earth itself, which is marked here and there with chains of mountains and depth of 
valleys. 


But Galileo was not satisfied with qualitative evidence. He famously stated that the 
book of nature is written in the language of mathematics, and he used mathematics, 
in the form of the Pythagorean theorem, along with some shrewd estimates, to assess 
indirectly the heights of lunar mountains. The process of indirect assessment is a 
hallmark of inverse problems in the natural sciences. (See [2] for an account of 
inverse problems of indirect assessment.) 

Nonuniqueness is a feature of many inverse problems that was slow to gain 
acceptance. An early instance of this phenomenon in a physical inverse problem 
occurred in the kinematic studies of ballistics carried out by Niccolo Tartaglia in 
the sixteenth century. Tartaglia claimed to be the inventor of the gunner’s square, a 
device for measuring the angle of inclination of a cannon. Using his square Tartaglia 
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carried out ranging trials and published some of the earliest firing tables. He studied 
not only the direct problem of finding ranges for a given firing angle but also 
the inverse problem of determining the firing angle that results in a given range. 
Although Tartaglia’s treatment was conceptually flawed and lacked rigor, his work 
contains glimmers of a number of basic principles of mathematical analysis that took 
several centuries to mature [32]. Tartaglia was particularly struck by nonuniqueness 
of solutions of the inverse problem. As he put it proudly in the dedication of his 
book Nova Scientia (Venice, 1537): 
I knew that a cannon could strike in the same place with two different elevations or aimings, 


I found a way of bringing about this event, a thing not heard of and not thought by any other, 
ancient or modern. 


With this boast Tartaglia was one of the first to call attention to this common feature 
of nonuniqueness in inverse problems. 

Tartaglia found that for a given fixed charge, each range (other than the maximum 
range) is achieved by two distinct aimings placed symmetrically above and below 
the 45° inclination. A century and a half later, Edmond Halley [39] took up the more 
general problem of allowing both the charge and the firing angle to vary while firing 
on a fixed target situated on an inclined plane. In this case the inverse problem of 
determining charge-angle pairs that result in a strike on the target has infinitely many 
solutions. (Of course, Halley did not address air resistance; his results are extended 
to the case of first-order resistance in [33].) Halley restored uniqueness to the inverse 
aiming problem by restricting consideration to the solution which minimizes what 
we would now call the kinetic energy of the emergent cannon ball. The idea of 
producing uniqueness by seeking the solution that minimizes a quadratic functional 
would in due course become a key feature of inverse theory. 

The model for a modern scientific society was laid out in Francis Bacon’s utopian 
novel The New Atlantis (1626). Bacon describes a voyage to the mythical land of 
Bensalem, which was inhabited by wise men inclined to inverse thinking. Solomon’s 
House, a research institute in Bensalem, was dedicated to the “knowledge of causes, 
and secret motions of things; and the enlarging of the bounds of human empire, 
to the effecting of all things possible” (my italics). The Royal Society of London, 
founded in 1660 and modeled on Baconian principles, was similarly dedicated. 
The first great triumph (due largely to Halley’s efforts) of the Royal Society was 
the publication of Newton’s magisterial Principia Mathematica (1687). In the 
Principia, Newton’s laws relating force, mass, and acceleration, combined with 
his inverse square law of gravity, were marshaled to solve the direct problem of 
two-body dynamics, confirming the curious form of Kepler’s planetary orbits: an 
inverse square centrally directed force leads to an orbit, which is a conic section. 
But Newton was not satisfied with this. He also treated the inverse problem of 
determining what gravitational law (cause) can give rise to a given geometrical orbit 
(effect). 

In the history of science literature, the problem of determining the orbit, 
given the law of attraction, is sometimes called the inverse problem; this practice 
inverts the terminology currently common in the scientific community. The reverse 
terminology in the history community is evidently a consequence of the fact that 
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Newton took up the determination of force law first and then treated the orbit 
determination problem. Indeed, Newton treated the inverse problem of orbits before 
he took up the direct problem. After all, his primary goal was to discover the laws 
of nature, the causes, rather than the effects. As Newton put it in the preface to the 
first edition of his Principia: “ ...the whole burden of philosophy seems to consist 
of this — from the phenomena of motions to investigate the forces of nature, and then 
from these forces to demonstrate the other phenomena.” 

In 1846, mathematical inverse theory produced a spectacular scientific triumph — 
the discovery of another world. The seeds of the discovery lay in the observed 
irregularities in the orbit of Uranus, the most distant of the planets known at 
the time. The orbit of Uranus did not fit with predictions based on Newton’s 
theories of gravity and motion. In particular an orbit calculated to fit contemporary 
observations did not fit observations made in the previous century, and an orbit that 
fit to the older sightings did not match the contemporary data. This suggested two 
possibilities: either Newton’s theory had to be modified at great distances or perhaps 
the anomalies in the orbit of Uranus were the effect of an as yet undiscovered planet 
(the cause) operating on Uranus via Newton’s laws. 

During the summer vacation of 1841, John Couch Adams, an undergraduate of 
St. John’s College, Cambridge, was intrigued by the second possibility. He recorded 
this diary entry: 

1841, July 3. Formed a design in the beginning of the week, of investigating, as soon 

as possible after taking my degree, the irregularities in the motion of Uranus, which are 

yet unaccounted for; in order to find whether they may be attributed to the action of an 


undiscovered planet beyond it; and if possible thence to determine the elements of its orbit, 
etc. approximately, which would probably lead to its discovery. 


Adams solved the inverse problem of determining the characteristics of the orbit of 
the undiscovered planet, now known as Neptune, that perturbs Uranus. However, 
a sequence of lamentable missteps, involving his own timidity, bureaucratic inertia, 
and other human factors, resulted in the honor of “discovering” the new planet on the 
basis of mathematics going to Urbain Le Verrier of France, who solved the inverse 
problem independently of Adams. This of course led to disappointment in England 
over the botched opportunity to claim the discovery and to a good deal of hauteur 
in France over the perceived attempt by the English to grab credit deserved by a 
Frenchman. The fascinating story of the unseemly squabble is well told in [36]. See 
also [63] for a recent update in which old wounds are reopened. 

Newton’s discussion of inverse orbit problems in his Principia, and vague doubts 
about the form of the gravitational force law raised prior to the discovery of 
Neptune, may have inspired other inverse problems. An early interesting “toy” 
inverse problem in this vein was published by Ferdinand Joachimstahl in 1861 
[47]. The problem Joachimstahl posed, and solved by an Abel transform, was to 
determine the law of gravitational attraction if the total force at any distance from a 
line of known mass density is given. 

Johann Radon laid the foundation of mathematical imaging science, without 
knowing it, in his 1917 memoir [61]. (An English translation of Radon’s paper may 
be found in [17].) Radon was concerned with the purely mathematical problem of 
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determining a real-valued function of two variables from knowledge of the values 
of its line integrals over all lines intersecting its domain. Although Radon evidently 
had no application in mind, his treatment was to become, after its rediscovery a 
half century later, the basis for the mathematics of computed tomography. (See 
[14] for more on the history of computed tomography.) Essentially the same result 
was obtained independently by Viktor Ambartsumian [1] who was interested in 
a specific inverse problem in astronomy. Proper motions of stars are difficult to 
determine, but radial velocities (relative to the earth) are obtainable from chromatic 
Doppler shift measurements. Ambartsumian used a mathematical model essentially 
equivalent to that of Radon to deduce the true three-dimensional distribution of 
stellar velocities from the distribution of the radial velocities. 

In the mid-1950s of the last century, Allan Cormack, a young physics lecturer at 
the University of Cape Town, who was moonlighting in the radiology department of 
Groote Schuur Hospital, had a bright idea. In Cormack’s words: 


It occurred to me that in order to improve treatment planning one had to know the 
distribution of the attenuation coefficient of tissues in the body, and that this distribution 
had to be found by measurements made external to the body. It soon occurred to me that 
this information would be useful for diagnostic purposes and would constitute a tomogram 
or series of tomograms, though I did not learn the word “tomogram” for many years. 


This was the birth of the mathematical theory of medical imaging. Cormack would 
not learn of Radon’s work for another two decades, but he developed the basic 
results for radially symmetric attenuation coefficient distributions and tested the 
theory with good results on a simple manufactured specimen in the form of a 
cylinder of aluminum encased in an annular prism of wood. The reconstructed 
piecewise constant attenuation function matched that of the known specimen well 
enough to show the promise of this revolutionary new imaging technology. 

In the 1990s, inverse thinking and indirect assessment led to another spectacular 
advance in astronomy: the discovery of extrasolar planets. Philosophers had specu- 
lated on the reality of planets linked to the stars at least since classical Greek times, 
and few in modern times doubted the existence of extrasolar planets. But convincing 
evidence of their existence had to await the development of sufficiently sensitive 
telescope-mounted spectrometers and the application of simple inverse theory. The 
indirect evidence of extrasolar planets consisted of spectral shift data extracted from 
optical observations of a star. 

In a single star—planet system, determining the variable radial velocity (relative 
to the earth) of a star wobbling under the gravitational influence of an orbiting 
planet of known mass and orbital radius is a simple direct problem — just equate 
the gravitational acceleration of the planet to its centripetal acceleration. (Consider 
only the simple case in which the planet, star, and earth are coplanar and the orbit 
is circular; an orbit oblique to the line of sight from earth introduces an additional 
unknown quantity. As a consequence of this obliquity, the relative mass estimated 
from the inverse problem is actually a lower bound for this quantity.) Using Doppler 
shift data, a simple inverse problem model may be developed for determining 
approximations to the relative planetary mass and orbital radius. The solution of the 
inverse problem enabled astronomers to announce in 1995 the existence of the first 
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confirmed extrasolar planet orbiting the star 51Pegasi. The millennia-old question 
of the existence of extrasolar worlds finally had a convincing positive answer. 

We bring this historical survey of inverse problems up to the present day with 
the greatest challenge in contemporary cosmology: the search for dark matter. 
Such matter, being “dark,” is by definition inaccessible to direct measurement. But 
recently an imaging model on the largest scale in the history of science has come to 
be used in attempts to assay this dark matter. The process of gravitational lensing, 
which is based on Einstein’s theory of curved space-time, presents the possibility 
of inverting the imaging model to estimate a dark mass (the gravitational lens) that 
intervenes between the observer on earth and an immensely distant light source. 
The dark mass warps space in its vicinity resulting, under appropriate conditions, 
in focusing onto the earth light rays that in flat space would not intersect the 
earth. In an extreme case in which the light source (e.g., a galaxy), the intervening 
gravitational lens (dark matter), and the collected image are collinear, this results 
in a phenomenon called an Einstein ring (first observed in 1979; see [22]). If the 
distances from earth to the lens and from the lens to source can be estimated, then 
the solution of an inverse problem gives an estimate of the dark mass (see [56]). 


3 Mathematical Modeling and Analysis 


p> ...we have to remember that what we observe is not nature in itself 
but nature exposed to our method of questioning. 
Werner Heisenberg 


A Platonic Inverse Problem 


Plato’s discussion of captives struggling to discern the real cause of shadows cast on 
the back wall of a cave is a primeval exemplar of inverse imaging problems. Here 
we present a toy imaging problem inspired by Plato’s allegory. While the problem is 
very elementary, it usefully illustrates some important aspects of imaging problems 
and inverse problems in general. 

Imagine a two-dimensional convex object in the x y-plane, which is bounded by 
the positive coordinate axes, and the graph of a function y = f(x),0 < x < 
1 that is positive on [0,1), strictly decreasing and concave-down, and satisfies 
fd) = 0 = f’(O). The object is illuminated by parallel light rays from the left 
that form angles 6 with the negative ray of the horizontal axis, as illustrated in 
Fig. 2. 

The goal is to reconstruct the shape of the object from observations of the 
extent s(@) of the shadow cast by the object. This is accomplished by fashioning a 
parameterization (x(@), f(x(@))) of the boundary curve of the object. As a further 
simplification we assume that f’(1) = —1. These assumptions guarantee that for 
each s > | there is a unique point (t, f(t)) on the graph of f at which the tangent 
line intersects the x-axis at s. What is required to see this is the existence of a unique 
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(x(8), f(x(®))) 


Fig. 2. A model shadow problem 


t € (0, 1) such that the tangent line to the graph at the point (t, f(t)) intersects the 
x-axis at s. That is, 


(s-—)f'O + fO =0. 
For each fixed s > 1, the expression on the left is strictly decreasing for t € (0, 1), 
positive at ¢ = 0 and negative at t = 1, so the existence of a unique such t = x(@) 
is assured. At the point of tangency, 
—tand = f’(x(@)). 
Also, 


F(x (8)) = (tan 8)(s(8) — x(@)), 


and hence determining x(@) also gives (x(@), f(x(@))), which solves the inverse 
imaging problem. Combining these results we have 


—(tan 6)x’(0) = f’(x(@))x'(0) = (s(@) — x(0)) sec? 6 + (s'(0) — x’()) tan 6. 
A bit of simplification yields 


x(0) = s(6) + : sin (20)s’(8), (1) 
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which explicitly solves the inverse problem of determining the shape (x(@), 
F(x(@))) from knowledge of the extent of the shadows s(6). 

The explicit formula (1) would seem to completely solve the inverse problem. 
In a theoretical sense this is certainly true. However, the formulation (1) shelters a 
subversive factor (the derivative) that should alert us to potential challenges involved 
in the practical solution of the inverse problem. Observations are always subject 
to measurement errors. The differentiation process, even if performed exactly, 
may amplify these errors as differentiation is a notoriously unstable process. For 
example, if a shadow function s(@) is perturbed by low-amplitude high-frequency 
noise of the form n, (0) = 1 sinn7@ giving observed data 


n 


Sn(8) = s(9) + mn (9), 


then the corresponding shape abscissas provided by (1) satisfy 


Xn (0) = (0) + 14() +r nh (8. 


But 7, converges uniformly to 0 as n — oo, while max|7),| — ov, giving a 
convincing illustration of the instability of the solution of the inverse problem 
provided by (1). For more examples of model inverse problems with explicit 
solutions that involve differentiation, see [71]. 

It is instructive to view the inverse problem from another perspective. Note that 
by (1), s(@) is the solution of the linear differential equation 


ds Py 2 2 
= x 
dd" sin(20) sin(20) 


(9) 


satisfying s(a/4) = 1. This differential equation may be solved by elementary 
means yielding 


s(0) = 


1+ cos20 9 2(1 + cos 20 
aa i ( x(y) dg. (2) 


sin 20 /4 (1 + cos 2¢) sin 20 


In this formulation, the “hidden” solution x(g) of the inverse problem is seen to be 
transformed by a linear integral operator into observations of the shadows s(0). The 
goal now is to uncover x(g) from (2) using knowledge of s(@), that is, one must 
solve an integral equation. 

The solution of the integral formulation (2) suffers from the same instability as 
the explicit solution (1). Indeed, one may write (2) as 


S=Wvt+wTx 
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where w(@) = (1 + cos26)/ sin26 and 


0 


2 
TIO = | omg )4e- 


If we let vn(g) = 5(1 + cos 2¢g) sin n’@ and set 
So =WtwT(x + vy) 


then one finds that s, — s uniformly, while max |v,| — oo. That is, arbitrarily 
small perturbations in the data s may correspond to arbitrarily large deviations in 
the solution x. This story has a moral: instability is intrinsic to the inverse problem 
itself and not a manifestation of a particular representation of the solution. 


Cormack’s Inverse Problem 


As noted in the previous section, the earliest tomographic test problem explicitly 
motivated by medical imaging was Cormack’s experiment [12] with a fabricated 
sample having a simple radially symmetric absorption coefficient. The absorption 
coefficient is a scalar field whose support may be assumed to be contained within 
the body to be imaged. This coefficient f supplies a measure of the attenuation 
rate of radiation as it passes through a given body point and is characterized by 
Bouguer’s law 


dl 
ee io oe 

ds f 

where J is the intensity of the radiation and s is arclength. The integral of f along 
a line L intersecting the body then satisfies 


c= | fas 


Here g = In(Jo/JI-), where Jo is the incident intensity, and J, the emergent 
intensity, of the beam. The observable quantity g is then a measure of the total 
attenuation effect that the body has on the beam traversing the line L. 

To be more specific, for a givent € R and a given unit vector @ = (cos@, sing), 
let L;¢ represent the line 


ijg=eR 6 .o=r 


where (-, -) is the Euclidean inner product. We will denote the integral of f over 
Lig by 
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Rife) = | fas= | f(tcosy —ssing,t sing + scosg) ds. 


7 


If f is radial, that is, independent of ¢, then 


R(F)(t.9) = R()(t.0) = / fiaie 3) 


Furthermore, if f vanishes exterior to the disk of radius R, then on setting r = 


Vt? +s? and f(r) = f(t,s), one finds 


27) 4 
t Vr ara 


where g(t) = R(f)(t, 0). The mapping defined by (4), which for a given line L, , 
transforms the radial attenuation coefficient into the function g, is an Abel transform 
of f. It represents, as a direct problem, Cormack’s early experiment with a radially 
symmetric test body. Determining the distribution of the attenuation coefficient 
requires solving the inverse problem. The Abel transform may be formally inverted 
by elementary means to furnish a solution of the inverse problem of determining the 
attenuation coefficient f from knowledge of the loss data g. Indeed, by (4) and a 
reversal of order of integration, 


gt) = (4) 


R tg(t) 
GESst- [of es 


since 


R 
dtds = xf F(s)s ds, 


dt=7 


S 
I Vs? — Vt? — Pr? 
(change the variable of integration to w = Vs? — t2/+/s2 — r2). However, 


R tg(t) 
r t2 _ r2 


R 
dt = -| (1? — r?)/2 9! (t) dt 


and hence on differentiating, we have 


© tea) 
Therefore, 
1% g(t) 
MO. eae 
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The derivative lurking within this inversion formula is again a harbinger of 
instability in the solution of the inverse problem. 


Forward and Reverse Diffusion 


Imagine a bar, identified with the interval [0, 2] of the x-axis, the lateral surface of 
which is thermally insulated while its ends are kept always at temperature zero. The 
diffusion of heat in the bar is governed by the one-dimensional heat equation 


ou ou 


—=Kk-~>, 0<x<a 
ot Ox? 


where u(x,t) is the temperature at position x and time ¢ and x is the thermal 
diffusivity. If the initial temperature distribution in the bar is a function f(x), then 
the boundary and initial conditions associated with this model are 


u(0,t) =0, u(z,t)=0, u(x,0) = f(x). 
In the forward diffusion problem, the goal is to find, for a given future time 


T > 0, the temperature distribution g(x) = u(x, T). Formal separation of variable 
techniques leads to a solution of the form 


lo.) 
ree 
u(x,t) = ) ane *" ' sinnx, 


n=1 


where a, are the Fourier coefficients of the initial temperature distribution 


2 IU 
an = =| f(s) sinns ds. 
a) 


The future temperature distribution is then seen to be, after some rearranging, 


g(x) = / k(x, 8) f(s) ds, 


where 


2 [o.@) 
-en2T : 
k(x,s)=— ) eT sinnx sinns. 
1 


n=1 


A high degree of smoothing is a notable feature of the forward diffusion process. 
Specifically, the factors e-*T in the transformation have the effect of severely 
damping high-frequency components in the initial temperature distribution /. 
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A corresponding reverse diffusion process is immediately suggested, namely, the 
retrodiction of the initial temperature distribution f, from knowledge of the later 
temperature distribution g. In this inverse problem, one finds that 


f(x) = 2 Stee r [ g(s) sinns ds. (5) 


aera) 


The contrast with the forward problem is striking: now high-frequency components 
in g are amplified by the huge factors eT _ Also note that the inverse problem is 
soluble only for a restricted class of functions g — those for which the series (5) 
converges in L7[0,z]. As will be seen in the next section, the reverse diffusion 
process is a useful metaphor in the discussion of deblurring. 


Deblurring as an Inverse Problem 


Cameras and other optical imagers capture a scene, or object, and convert it into 
an imperfect image. The object may be represented mathematically by a function 
f : R? = R that codes, for example, gray scale or intensity. The image produced 
by the device is a function g : R? — R, and the process may be phrased abstractly 
as g = Kf, where K is an operator modeling the operation of the imager. In a 
perfect imager, K = J, the identity operator (recall Mein Herr’s map!). The perfect 
model may be expressed in terms of the two-dimensional delta distribution as 


r= f se-bsreae. 


However, any physical imaging device blurs the object f into an image g, which in 
many cases can be represented by 


a= ff kG-Dseaé ©) 


where k(-), the point spread function of the device, is some approximation of the 
delta function centered at the origin. Theoretical examples of such approximations 
include the tin-can function yr/(2R*), where yz is the indicator function of the 
disk of radius R centered at the origin, and the sinc and sombrero functions given 
in polar coordinates by 


i J 
mn and somb(r, 0) = 2——— a 


sinc(r, 0) = 


respectively, where J; is the Bessel function of first kind and order 1. A frequently 
occurring model uses the Gaussian point spread function 
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1 2 
k(r, 0) = a ev [207 


The problem of deblurring consists of solving (6) for the object f, given the blurred 
image g. For a good introduction to deblurring, see [44]. 

Reverse diffusion in two dimensions is a close cousin of deblurring. A basic tool 
in the analysis is the 2D Fourier transform defined for f : R* > R and x,@ € 
R? by 


n~ oo oo jo 
7@)=FI@ =f fF f@)dmars 
—oo J—0o 
with the inversion formula 
Lf” [% earz 
f(x) = J i e'@! £(@) daw, dw. 
(2%)? J—sod oo fo) 

On integrating by parts, one sees that 


F{Af}(G) = —||6I|? FG), 


; : 2 2 : ys 
where A is the Laplacian operator: A = a + oy. Consider now the initial value 
sak w2 


problem for the 2D heat equation 


a =xAu X€R’, t>0, u(X,0) = F(X). 


Applying the Fourier transform yields the initial value problem 


dU i ‘ 
Ge lel U, UO)=f 


where U(t) = u(-,t) and hence U(t) = Fe lellr, The convolution theorem then 
gives 


ut) = FYerloPerZy — if . / ” KB) @ db dé 


where (using the integral result of [25], 12.A) 


k(X) 


—1 J -o2kt ,—-wsKt 1 = i(X,0) —(w? +03 )xet 
F {e KT e 9 jaa eb Ole Vr rr)eT day, day 
—oCo —oCo 


eo GI+33)/Aet) ; 
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The inverse problem of determining the initial distribution f(x) = u(X,0), given 
the distribution g(x) = u(x, T) at a later time T > 0, is equivalent to solving the 
integral equation of the first kind 


1 
4nxT 


g(x) = [ [- eo (Gry +28) ET) FE, £5) dE, dbo, 


which is in turn equivalent to the deblurring problem with Gaussian point spread 
function 


BS 1 =I I2 /202 
To(%) = =e II /2 


having variance o* = 2xT.. The idea of deblurring by reverse diffusion is developed 
in [8]. 


Extrapolation of Band-Limited Signals 


Extrapolation is a basic challenge in signal analysis. The Fourier transform, /, is the 
analytical workhorse in this field. It transforms a time signal f(t),-oo < t < ©, 
into a complex-frequency distribution f (w) via the formula 


Flo) = FLV) = i] fe at. 


In a suitable setting, the time-to-frequency transformation may be inverted by the 
formula (e.g., [25]) 


fO= FO =e / - Flwjel dt. 


Any physically realizable detector is capable of picking up frequencies only in a 
limited range, say |w| < Q. A signal f whose Fourier transform vanishes for |w| > 
Q, for some given Q > 0, is called a band-limited signal. A detector that operates 
in the frequency band —Q < w < Q band-limits signals it collects, that is, it treats 


only X|-2,9),f, where 


1, w € [-Q, Q] 
X-2.2) =) 9 ¢ [-2.9] 
is the indicator function of the interval [—Q, Q]. Multiplication by y{-g,q] in the 
frequency domain is called a low-pass filter as only components with frequency 
|w| < Q survive the filtering process. 
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Reconstruction of the full signal f is generally not possible as information in 
components with frequency greater than Q is unavailable. What is available is the 
signal 


g=F {yeaah}. 


By the convolution theorem for Fourier transforms, one then has 


g=F yon} f 
However, 


1 a sin Qt 
F yon) = = / etdw = . 
wh 


-Q mt 


The reconstruction (or extrapolation) of the full signal f given the detected signal 
g requires the solution of the convolution equation 


g(t) = [ em) pais. 


loo) 8 (¢ ~ T) 
The problem of extrapolating a band-limited signal is then seen to be mathematically 
the same as deblurring the effect of an instrument with the one-dimensional point 
spread function 


PET 


CT scanning with X-rays is an instance of transmission tomography. A decade and 
a half prior to Cormack’s publications on transmission tomography, an emission 
tomography technique, now known as PET (positron transmission tomography), 
was proposed [72]. In PET, a metabolically active tracer in the form of a positron- 
emitting isotope is injected into an area for which it has an affinity and taken up 
(metabolized) by an organ. The isotope emits positrons that immediately combine 
with free electrons in so-called annihilation events, which result in the ejection 
of two photons (y-rays) along oppositely directed collinear rays. When a pair of 
detectors located on an array surrounding the body pick up the simultaneous arrival 
of two photons, one at each detector, respectively, an annihilation event is assumed 
to have taken place on the segment connecting the two detectors. In PET, the data 
collected from a very large number of such events is used to construct a two- 
dimensional tomographic slice of the isotope distribution. Because the uptake of the 
isotope is metabolically driven, PET is an effective tool for studying metabolism 
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giving it a diagnostic advantage over X-ray CT scanning. A combination of an 
X-ray CT scan with a PET scan provides the diagnostician anatomical information 
(distribution of attenuation coefficient) and physiological information (density of 
metabolized tracer isotope), respectively. 

If f : R? — R (we consider only a simplified version of 2D-PET) is the density 
of the metabolized tracer isotope, then the number of annihilations occurring along 
the coincidence line L connecting two detectors is proportional to the line integral 


[ fas. 


That is, the observed counts of annihilation events are measured by the Radon 
transform of the density {. However, this does not take account of attenuation 
effects and can under represent features of deep-seated tissue. If the attenuation 
distribution is j1(-, -), and the pair of photons resulting from an annihilation event 
on the coincidence line L traverse oppositely directed rays L+ and L_ of L, 
emanating from the annihilation site, then the detected attenuated signal takes the 
form 


g= / Pan Oe Te Hee Pls 
L 


e~ IL udu / fds. 
L 


The model operator may now be viewed as a bivariate operator K(u, f) = g, in 
which the operator K(-, f) is nonlinear and the operator K(,1, -) is linear. In soft 
tissue the attenuation coefficient is essentially zero, and therefore the solution of 
the inverse problem is accomplished by a Radon inversion of K(0, -). PET scans 
may be performed in combination with X-ray CT scans; the CT scan provides the 
attenuation coefficient, which may then be used in the model above to find the 
isotope density. See [52] for an extensive survey of emission tomography. 


Some Mathematics for Inverse Problems 


> Philosophy is written in that great book which ever lies before our 
gaze — | mean the universe .... The book is written in the 
mathematical language ... without which one wanders 
in vain through a dark labyrinth. 
Galileo Galilei 


Hilbert space is a familiar environment that is rich enough for a discussion of the 
chief mathematical issues that are important in the theory of inverse problems. For 
the most part we restrict our attention to real Hilbert spaces. The inner product and 
associated norm will be symbolized by (-, -}) and || - ||, respectively: 
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IIx ll = V(x, x). 


We assume the reader is familiar with the basic properties of inner product spaces 
(see, e.g., [18, Chap. I]), including the Cauchy—Schwarz inequality 


I(x, y)] < Illy 


A Hilbert space H is complete, that is, Cauchy sequences in H converge: 


if lim ||x,—-—X»||=0, then ||x, —x|| > 0, 
n,m Oo 
for some x € H. The smallest (in the sense of inclusion) Hilbert space that contains 
a given inner product space is known as the completion of the inner product space. 
(Every inner product space has a unique completion.) 
The space, denoted L”[a, b], of measurable functions on an interval [a, b] whose 
squares are Lebesgue integrable, with inner product 


b 
Gas i: Fglt) dt, 


is the prototypical example of a Hilbert space. The Sobolev space of order m, 
Ha, b], is the completion with respect to the norm 


m 1/2 
If llo = (>: i] ; 


k=0 
associated with the inner product 


m 


(f &)m a Sor 2 la 


k=0 


of the space of functions having m continuous derivatives on [a, b]. Here (-, -)9 and 
|| « lo are the L?[a, b] norm and inner product; of course, H [a,b] = L*[a, 5]. 

Two vectors x and y ina Hilbert space H are called orthogonal, denoted x L y, 
if (x, y) = 0. The Pythagorean Theorem, 


x Ly => |x + yl? = [xll? + Uy? 
is a key notion that suggests the transfer of familiar geometrical ideas from 
Euclidean space to Hilbert space. The orthogonal complement of a set S is the 


closed subspace 


St={xeH:x 1s, forall s eS}. 
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It is not difficult to show that if S is a subspace, then S tt = §, where S is the 
closure of S, that is, the smallest closed subspace that contains S. A closed subspace 
S of a Hilbert space H engenders a Cartesian decomposition of H, symbolized by 
H = S @ S+, meaning that each x € H has a unique representation of the form 
xX = xX; + Xo, where x; € S is the projection of x onto S: 


[|x — x1|| = inf {lx — yl]: y € S}, 


and similarly x2 is the projection of x onto S+. The projection of a vector x onto a 
closed subspace S is denoted by Psx. 

A set of mutually orthogonal vectors each of which has unit norm is called an 
orthonormal set. An orthonormal set S is complete if St = {0}. A complete 
orthonormal system for a Hilbert space is a sequence of vectors in H, which is 
complete and orthonormal. For example, {sinnat :n = 1,2,3,...} is a complete 
orthonormal system for the Hilbert space L[0,1]. Each vector x € H has a 
convergent Fourier expansion in terms of a complete orthonormal system {@,}°° , 
for H: 


CO 


x= S (x, Pn) Gns 


n=1 


of which Parseval’s identity is an immediate consequence 


lll? = S01, a). 


n=1 


Weak Convergence 

“Weak” notions are crucial to our development of mathematics for inverse problems. 
Suppose the class of functions of interest forms a real Hilbert space H with an 
inner product (-, -) and associated norm || - ||. A functional is a mapping from H 
to R. It is helpful to think of a functional as a measurement on elements of H. 
Proportionality and additivity are natural features of most measuring processes. A 
functional F : H — R with these features, that is, satisfying 


F(ax + By) = aF(x) + BF(y) 


where @ and f are scalars and x,y € H, is called a linear functional. Another 
common and highly desirable feature of a measurement process is continuity: 
elements of H which are nearly the same should result in measurements that are 
nearly the same. In mathematical terms, a functional F is continuous if, as 1 > o0, 


|x, —x|| > 0 implies |F(x,) — F(x)| > 0. 
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For example, if the Hilbert space is L?[0, 7], the space of square integrable functions 
on [0, 7], then the average value functional, 


T 
F(x) = =f x(t) dt, 


is a continuous linear functional. (This is an immediate consequence of the Cauchy— 
Schwarz inequality.) 

The Riesz Representation Theorem characterizes continuous linear functionals 
on a Hilbert space: 


A continuous linear functional F on H has a unique representation of the form 


F(x) = (x,9) 
for some g € H. 


This result is so fundamental that it is worthwhile to sketch a micro-proof. We may 
assume that F is not identically zero (otherwise, take g = 0), and hence there is a 
z € H, with F(z) = 1, which is orthogonal to the closed subspace 


N={x € A: F(x) = 0}. 
Then x — F(x)z € N forall x € H, and hence g = z/||z||* fits the bill 


O = (x — F(x)z,2/llzl?) = (x, 9) — FQ). 


Any two distinct vectors in H are distinguishable by some measurement in the 
form of a continuous linear functional. Indeed, if (x — y,g) = 0 for all g € H, 
then x = y (set g = x — y). However, it is possible for a sequence of vectors 
{xn}, which does not converge in H to any vector, nevertheless to be ultimately 
indistinguishable from some vector x by bounded linear functionals. This is the idea 
of weak convergence. We say that {x, } converges weakly to x, symbolized x, — x, 
if (x,,~) — (x,@) for every g € H. The simple identity 


lx = xXnll? = (x = Xn, = Xn) = [el]? + [lnll? — 20m, x) 


shows that if x, — x and ||x,|| — ||x||, then x, converges strongly to x, that is, 
\|Xn — x|| > 0. 

In a Hilbert space, every sequence of vectors whose norms are uniformly 
bounded has a subsequence that is weakly convergent (e.g., [18], p. 205). We 
note that any complete orthonormal system {¢,} converges weakly to zero, for by 
Parseval’s identity 


D1, Gn)? = Hlell?, 


and hence (x, @,) — 0asn — oo foreach x € H. 
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A set is called weakly closed if it contains the weak limit of every weakly 
convergent sequence of vectors in the set. Hilbert spaces have the following feature 
(see, e.g., [30]), which is fundamental in the theory of optimization: 


Suppose C is a weakly closed convex subset of a Hilbert space H. For each x € H, there 
is a unique vector Pc(x) € C such that 


Ix — Pc(x)|] = inf{]|x — ul] sw € Ch. 


Pc(x) is called the metric projection of x onto C. It can be shown that a closed 
convex set is also weakly closed. 


Linear Operators 
A bounded linear operator from a Hilbert space Hj into a Hilbert space Hp is a 
mapping K : H,; — Hb, which is linear, K(ax + By) = aKx + BKy, and for 
which the number 

|| Kl] = sup {|| Kx||/[lxl] : x 4 OF 


is finite. Note that we have used the same symbol for the norm in each of the spaces; 
this generally will be our practice in the sequel. If K is a bounded linear operator, 
then K is (uniformly) continuous since 


||Kx — Ky|| = |K@—y)Il s IKIlllx — yl. 


For our purposes, the most cogent example of a bounded linear operator is an 
integral operator K : L?[a,b] > L*[c, d] of the form 


b 
Kf = | k(t,s) f(s)ds, c<t<d, (7) 


where k(-, -) € L?([c, d] x [a, b]) is called the kernel of the integral operator. The 
kernel is called degenerate if it has the form 


m 


k(t,s) = )- T)()S;(s) 


j=l 


where the 7; and the S; are each linearly independent sets of functions of a single 
variable. In this case the range, R(K), of the operator K is the finite-dimensional 
subspace 

R(K) = span{T; : j =1,...,m} 


and 
m 


Kf (t) = Yotk(t, +), fT). 


j=l 


where (-, -) is the L?[a, b] inner product. 
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The adjoint of a bounded linear operator K : H; — Hz is the bounded linear 
operator K* : Hy — Hj, which satisfies 


(Kx, y) = (x, K*y) 


for all x € H, and y € Ay. For example, by changing the order of integration, one 
can see that the adjoint of the integral operator (7) is 


d 
Kgs) = | k(u, s)g(u) du. 


A bounded linear operator K : H — A is called self-adjoint if K* = K.The 
null-space of a bounded linear operator K : H; — Hz is the closed subspace 


N(K) = {x € HM, : Kx = 0}. 
Note that N(K*K) = N(K) for if x € N(K*K), then 
0 = (K*Kx,x) = (Kx, Kx) = ||Kx|/’. 
There are fundamental relationships between the null-space and the range 
R(K) = {Kx:x € Hi}, 
of a linear operator and its adjoint. In fact, y € R(K)+ if and only if 
0 = (Kx, y) = (x, K*y) 
for all x € H,, and hence R(K)+ = N(K*). It follows that R(K) = R(K)1+ = 


N(K*)+. Replacing K by K* in these relations (noting that K** = K), we obtain 
the four fundamental relationships: 


R(K)* = N(K*), R(K) = N(K*)* 
R(K*)+ = N(K), R(K*) = N(K)t. 


Examples in a previous section have highlighted the unstable nature of solutions 
of inverse problems. This instability is conveniently phrased in terms of linear 
operators that are unbounded. Unbounded linear operators are typically defined only 
on restricted subspaces of the Hilbert space. For example, L7[0, zr] contains discon- 
tinuous, and hence nondifferentiable, functions. But the differentiation operator may 
be defined on the proper subspace of L7[0, z] consisting of differentiable functions 
with derivatives in L7[0, 1]. This differentiation operator is unbounded since 
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while 
d(1. 5 2 5 
—|{-—sinn*t =—n—>o as low. 
dt \n 2 


The reverse diffusion process is also governed by an unbounded operator. As seen 
in (5), the operator L, which maps the temperature distribution g(x) = u(x, T) to 
the initial temperature distribution f(x) = u(x, 0), is defined on the subspace 


Co 
ge L7[0,2]: > eT |b, |? < cof , 


n=1 


D(L) = 


where 
2. {[* ; 
by, = = [ g(s) sinns ds. 
HT Jo 


L is unbounded because the functions ¢,,(s) = sinms reside in D(L) and satisfy 
|G ||? = 2/2, but by the orthogonality relationships, 


eknT 


LOn = Pm, 


and hence | L@m ||? = e2kn?T 


m/2—> casm > oo. 
Compact Operators and the SVD 
A bounded linear operator K : H; — H)? of the form 


r 


Kx = )°(x,v,)uj, (8) 


j=l 


where {u; };=1 is a linearly independent set of vectors in Hz and {v;}/,_, is a set 
of vectors in Hj, is called an operator of finite rank (with rank = r). For example, 
an integral operator on L*[a,b] with a degenerate kernel is an operator of finite 
rank. Finite rank operators transform weakly convergent sequences into strongly 
convergent sequences: if x, — x, then 


r 


Kx, = Drm vy) uj > Sox, vy)uy = Kx. 


j=l j=l 
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More generally, a linear operator is called compact if it enjoys this weak-to-strong 
continuity, that is, if x, —- x implies that Kx, — Kx. In terms of our metaphor 
of bounded linear functionals as measurements, one could say that if the linear 
operator K modeling an inverse problem is compact, and if all measurements on a 
sequence of functions {x,} are ultimately indistinguishable from the corresponding 
measurements on x, then the model values Kx, are ultimately indistinguishable 
from Kx. That is, causes, which are ultimately indistinguishable by linear measure- 
ment processes, result in effects that are ultimately indistinguishable. It is therefore 
not surprising that compact operators occur frequently in models of linear inverse 
problems. 

Erhard Schmidt’s theory of singular functions, now called the singular value 
decomposition (SVD), is the most versatile and effective tool for the analysis of 
compact linear operators. (The SVD has been rediscovered several times in various 
contexts; for the curious history of the SVD, see [65].) The SVD extends the 
representation (8) in a particularly useful way. A singular system {vj;, uj; [Wj ei 
for a compact linear operator K bundles a complete orthonormal system {v pst 
for N(K)+, consisting of eigenvectors of K* K; a complete orthonormal system 
{uj }2, for N(K*)+ = R(K), consisting of eigenvectors of K K*; and a sequence 
of positive numbers j;, called singular values of K. The singular values and 
singular vectors are tied together by the relationships 


Kvj = pju;, K* a; = pyv;, f = 1,23 35088 


Every compact linear operator has a singular system, and the action of the operator 
may be expressed in terms of the SVD as 


00 
Kx = 0 pj x,vj)uy (9) 
j=l 
In case K has finite rank r, this sum terminates at 7 = r, and otherwise 
fj 270, as jf > Om. 
We shall see that this fact is singularly important in the analysis of inverse problems. 


As an example of the SVD of a nonself-adjoint compact operator, consider the 
integral operator K : L?[0, 2] > L?[0, x] defined by 


(Kf)(t) = i " n(t.u) flu) du 


where 


10<u<t 
O,t<u<7. 


A(t,u) = 
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One can verify that a singular system {v;, uj; [Wj be , for this operator is 


vy(0) = cos Mase u;(s) = cae a a es 
ING a) ee - § oa 


A compact operator has closed range if and only if it has finite rank. This follows 
from the SVD and the open mapping theorem (e.g., [18], p. 166). Indeed, if K is 
compact and R(K) is closed, then the restricted operator K : N(K)+ > R(K) is 
one-to-one and onto, and hence has a bounded inverse. That is, there is a positive 
number m such that || Kx|| > m||x|| for all x € N(K)+. But then, by (9), 


Lj = bj \lujll = | Kyl] 2 mllvj|]| =m > 0, 


and hence K has only finitely many singular values for otherwise 4; — 0. This 
result is highly significant in inverse theory for it says that finite rank linear models, 
when pushed too far toward the limiting case of an operator of infinite rank, will 
inevitably result in instability. 

In 1910, Emil Picard [60] established a criterion that characterizes the existence 
of solutions of an equation of the first kind 


Kx=y, (10) 
where K is acompact linear operator. Picard’s criterion plays a role in inverse theory 
analogous to that which the Fredholm alternative plays for integral equations of the 
second kind. 

Since {v;} is a complete orthonormal system for N(K )+, the series 
Co 
Dele vAP 
j=l 


is convergent (and equals | Prvcxyt xl”). However, if y = Kx € R(K), then 


(x, vj) = wy? (x, K*uj) = wy'(Kx, uj) = wy" (y, uy), 


and so 


[oe 
S*172Iky.0))I2 < 00 


j=l 


is a necessary condition for y € R(K). 
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On the other hand, this condition guarantees that the series 


m= > py yj); (11) 


j=l 


is convergent in R(K)+ and the singular value relations show that Kx = Pye) y- 
Taken together these results establish the Picard Criterion: 


ye R(K) @yeN(K*)+ and SY °y7?|(y, uj)? < 00. (12) 


j=l 
If y satisfies Picard’s criterion, then y = Kx where x is given by (11). 


The Moore-Penrose Inverse 

If y ¢ R(K), then the equation Kx = y has no solution, but this should not prevent 
one from doing the best one can to try to solve the problem. Perhaps the best that 
can be done is to seek a vector u that is as near as possible to serving as a solution. 
A vector u € H, that minimizes the quadratic functional 


F(x) = ||Kx— yl? 


is called a least squares solution of Kx = y. It is not hard to see that a least 
squares solution exists if and only if y belongs to the dense subspace R(T) + R(T)+ 
of Hz. Also, as the geometry suggests, u is a least squares solution if and only if 
y — Ku € R(K)+ = N(K*), and hence least squares solutions are vector v that 
satisfy the so-called normal equation 


K*Kv= K*y. (13) 


Furthermore, the solution set of (13) is closed and convex and therefore contains 
a unique vector nearest to the origin (i.e., of smallest norm), say v'. This smallest 
norm least squares solution v’ lies in N(K)+, for otherwise Pv' 4 0, where P is 
the orthogonal projector onto N(K). The Pythagorean theorem then gives 


lv? = vt — Pvt? + Pvt. 


But, since K* KPy' = 0, this implies that v' — Pv" is a least squares solution with 
norm smaller than that of v’. This contradiction ensures that v' € N(K)t. 

The operator Kt : D(K*) — N(K)+, which associates with each y in the dense 
subspace D(K*) = R(K)+ R(K)* of Hp the unique minimum norm least squares 
solution K'y € N(K)+ of the equation Kx = y, is called the Moore—Penrose 
generalized inverse of K.(E. H. Moore died when Roger (now Sir Roger) Penrose 
was an infant; [5] tells the story of how the names of both men came to be associated 
with the generalized inverse.) 
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If K is a compact linear operator with SVD {v;,u;;;} and y € D(K‘*), then 
the vector 


fo) 


> F(yuj)v, 


=o 


is well defined by Picard’s criterion, resides in N(K)+, and is a least squares 
solution. Therefore, 


CO 


1 
Ky= > Pe (14) 


=i 


The operator K ¥ so defined is linear, but it is unbounded (unless K has finite rank) 
since 


\lun || = 1, but ||Ktu,|| =1/un,—2>0o as n> oo. (15) 
There is an immense literature on generalized inverses; see [54] for a start. 


Alternating Projection Theorem 

Draw a pair of intersecting lines. Take a point at random in the plane spanned by the 
lines, and project it onto the first line. Then project that point onto the second line, 
and continue in this manner projecting alternately onto each line in turn. It soon 
becomes apparent that the sequence of points so generated zigzags and converges 
to the point that is common to both lines. In 1933, von Neumann showed the same 
behavior for two closed subspaces S; and S> of a Hilbert space H. Namely, for each 
xe dH, 


(Ps, Ps,)"x > Ps,ns;X aS n> ono, 


where Py stands for the orthogonal projector onto the closed subspace W. This 
result extends easily to the case where S, and S> are translates of closed subspaces, 
that is, closed affine sets (see, e.g., [18,35] for proofs). In fact, a modification 
of the method, due to Boyle and Dykstra (see [18], p. 213), provides a sequence 
that converges to the metric projection onto the intersection of a finite collection of 
closed convex sets. 

Stefan Kaczmarz [48] developed an alternating projection algorithm, indepen- 
dently of von Neumann, for approximating solutions of underdetermined systems 
of linear algebraic equations. (See [57] concerning Kaczmarz’s early and tragic 
demise.) A solution X € R” of a system of m linear equations in n unknowns with 


coefficient matrix A and right-hand side b lies in the intersection of the hyperplanes 


mw; = {x ER": (4;,X) = bj}, i= 1,2,...,m, 
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where G; is the ith row vector of A. Kaczmarz’s algorithm, which consists of 
successively and cyclically projecting onto these hyperplanes, produces a sequence 
of vectors that converges to that vector in the intersection of the hyperplanes, 
which is nearest to the initial approximation (see [20] for a complete treatment 
of the method). Kaczmarz’s method has come to be known as ART (the algebraic 
reconstruction technique) in the tomography community. 


4 Numerical Methods 


> All of exact science is dominated by 
the idea of approximation. 
Bertrand Russell 


Tikhonov Regularization 


The unboundedness of the operator K*, displayed in (15), is a fundamental 
challenge when solving linear inverse problems of the form Kx = y. This 
unboundedness is manifested as instability when the data vector y contains errors, 
which is always the case in practical circumstances as the data result from obser- 
vation and measurement. Small errors in high-order singular components (y, u,) (” 
large) will be magnified by the factor 1/1, in the representation (14), resulting in 
large deviations in the computed solution. Such instabilities in numerical solutions 
were noticed from the very beginning of the use of digital computers to solve 
linear inverse problems (see [31] for examples and references). The development 
of theoretical strategies to mitigate this instability is known as regularization 
theory. 

One way to stabilize the solution process is to restrict the notion of solution. 
Tikhonov’s classic result [66] of 1943 is an instance of this idea. In that paper 
Tikhonov treated the inverse problem of determining the spatial distribution of a 
uniform star-shaped mass lying below the horizontal surface from measurements 
of the gravitational potential on the surface. He showed that the inverse problem 
becomes well posed if the forward operator is restricted to a certain compact set. 
Another approach is to modify the forward operator itself without a restriction on 
its domain. In what has come to be known as Tikhonov regularization, the notion 
of solution is generalized to the minimum norm least squares solution, which is 
unstable, but a stable approximation to this generalized solution, depending on a 
regularization parameter, is constructed. 

The idea of Tikhonov regularization may be introduced from either an algebraic 
or a variational viewpoint. Algebraically, the method, in its simplest form, consists 
in replacing the normal equation (13) with the second kind equation 


K*Kv+av= K*y, (16) 
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where aq is a positive parameter. The key point is that the problem of solving (16) is 
well posed. Indeed, 


|x |K*K + aT || > ((K*K +a1)x,x) = ||Kx|P? + ax? > oll), 


and hence (K* K +a/)~! is a bounded linear operator; in fact, ||(K*K +aI)~'|| < 
1/a. The significance of this fact is that for fixed a > 0, the approximation 


Xy = (K*K +al) 1 K*y (17) 
depends continuously on y. Specifically, suppose y® is an observed version of 
y satisfying ||y — y®|| < 6, and let x5 be the approximation formed using this 
approximate data, that is, 

—_— * —-1p7*,,65 

xy = (K*K+al) K*y°. 


From the SVD we have 


CO 
LL 
Xg—x8 = y ; dl (y—y%,uj)v;, 
j 


and hence 


co 


ey igs 
Hi +O py +o 


m (18) 
< EDR lly — y's)? < 8 /a. 


If the minimum norm least squares solution K‘y satisfies the source condition 
Kty = K* Kw, for some w € Hj, then one can show that 


Kty —Xy =a(K*K +al)!'K*Kw 


and hence 


a 2 
Kiy— 2 _ ai (2 < 92 2 19 
|Kty — xal? = 0? > ica) Koved? sop (19) 
j=l J 


Combining this with (18), we see that if K'y € R(K*K), then 


xi — Kty|] <8/Vat O(0). 
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Therefore, an a priori choice of the regularization parameter of the form 
a =a(8) = C8, (20) 
yields a convergence rate of the form 


|= = Kty| = 0(87/). (21) 


This two thirds power rate is an asymptotic “brick wall” for Tikhonov regularization 
in the sense that it is impossible to uniformly improve it to a 0(67/*) rate unless the 
compact operator K has finite rank (see [27]). Roughly speaking, this says that 
the best one can hope for is 2m-digit accuracy in the solution if there is 3m-digit 
accuracy in the data. 

The Tikhonov approximation (17) has a variational characterization that is useful 
in both theoretical analysis and computational implementation. The equation (16) 
that characterizes the Tikhonov approximation is the Euler equation for the func- 
tional 


Fa(-3y) = K+ — yl? +a [?, (22) 


and hence the approximation (17) is a global minimizer of (22). This opens 
the possibility of applying standard optimization techniques for calculating the 
Tikhonov approximation. Next we illustrate the usefulness of the variational 
characterization in a convergence analysis for an a posteriori selection technique 
for the regularization parameter known as Morozov’s discrepancy principle. 

The a priori parameter selection criterion (20) is of theoretical interest as it gives 
information on the order of magnitude of the regularization parameter that can be 
expected to result in a convergent procedure. However, a posteriori methods of 
choosing the regularization parameter that depend on the actual progress of the 
computations would be expected to lead to more satisfactory results. Morozov’s 
discrepancy principle [53] is the earliest parameter choice strategy of this type. 
Morozov’s idea (which was presaged by Phillips [31,59]) is to choose the regu- 
larization parameter in such a way that the size of the residual | K x — 2 | is equal 
to error level in the data: 


|Kx2-y? | =8. (23) 


It should be recognized that this condition contains some “slack” as 6, the bound for 
the data error, might not be tight. Nevertheless, this choice is not only possible, but 
it leads to a convergent procedure, as we now show. 

If ||y°|| > 6, that is, there is more signal than noise in the data, and if y € R(K), 
then there is a unique positive parameter a satisfying (23). To see this, we use the 
SVD 
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se 2 
2 a 
| Kx — P= D0 | a J Mo? ws P + Py? IP (24) 
j=l ype 
where P is the orthogonal projector of Hz onto R(K)+. From this we see that the 
real function 


Va) = ||Kxa— y' | 
is a continuous, strictly increasing function of @ satisfying (since Py = 0) 
lim, W(@) = |Py"l| = Pg’ — Pail < lly’ —yll <8 


and 


im ya) = lly > 6. 


The intermediate value theorem then guarantees a unique a = @(6) satisfying (23). 
We now show that the choice a (6) as given by the discrepancy method (23) leads 
to a regular scheme for approximating K'y: 


x8) —> Kty as 5>0. 


To this end it is sufficient to show that for any sequence 6, —> 0, there is 
a subsequence, which we will denote by {6,}, that satisfies oe —> Kty. 
The argument relies on the following previously discussed facts: norm-bounded 
sequences contain a weakly convergent subsequence, and weak convergence along 
with convergence of the norms implies strong convergence. 

We assume that y € R(K) and that K : H,; — Hz is a compact linear 
operator, and we let x = K‘y. That is, x is the unique vector in N(K)+ satisfying 
Kx=y. 

The variational characterization of the Tikhonov approximation xa) as the 


global minimizer of the quadratic functional F,(-; y°) (see (22)) implies that 


8 avg #430 
Fas) (xb ) S Fay; y"), 
that is, 


2 2 2 
$+ 0(8) [x4] =| Kxe—»*] +2) | 5) 


A 


Fas)(x) = lly — y? ||? + (8) 01? 


IA 


5° + a(5)||2|? 
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and hence |= | < ||x||. Therefore, for any sequence 6, — 0, there is a 


subsequence 6, — 0 with ie we for some w. But 
xo) = K*(KK* + a(5)1)|y® € R(K*) © M(K)~ 


and N(K)+ is weakly closed, and so w € N(K)+. Furthermore, 


Ok Sk 
| Kx) —¥" 


—>0 


and hence K: is — y. But as K is compact, K Pe — Kw, and it follows that 


Kw = y andw € N(K)+, that is, w = x. Since | 


ce | < ||x||, we then have 


2 : 8x : 
x|/- = lim (x 22 lim | 
Jo? = im (xehg,)-%) Slim 
and therefore 


. 5 7. 
xl < tim _[x2%60| < Tim roo | 


5 
tea) lk 


— ||x||, and hence tas > x, 


be 5k 
So we have shown that Xev(Sp) x and | Xav(5x) 


completing the proof. 

It can be shown that, under the source condition x € R(K*), Tikhonov’s method 
with parameter choice by the discrepancy principle (23) achieves an asymptotic 
order of accuracy O(/8); however, a swifter rate of o(V/8) is generally impossible 
except in the case when K has finite rank [26]. Engl and Gfrerer (see [19, Chap. 4]) 
have developed a modification of the discrepancy principle that achieves the optimal 
order of convergence. 

Our sketch of the basic theory of Tikhonov regularization has assumed that 
the regularization functional, which augments the least square objective functional 
||K - — y||?, is (the square of) a norm. (Note however that while the same symbol 
is used for the norm in each of the spaces H; and Hp, these norms may be distinct. 
In his original paper [67], Tikhonov used a Sobolev norm on the solution space and 
an L? norm on the data space.) Phillips [59], in a paper that barely predates that of 
Tikhonov, used a regularizing semi-norm — the L? norm of the second derivative. 
In all of these cases, the equation characterizing the regularized approximation is 
linear. However, certain non-quadratic regularizing functionals, leading to nonlinear 
problems for determining the regularized approximation, are found to be effective 
in imaging science. Of particular note is the total variation, or TV, functional: 


TV) = ff Va 
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where wu: Q C R” —> R. Regularization now consists of minimizing the augmented 
least squares functional 


Fy(u) = || Ku _ Vi@ +aTV(u), 


where K is the identity operator for denoising problems, while for deblurring 
problems, K is the blurring operator associated with a known point spread function. 
A full exposition may be found in [62]. 


Iterative Regularization 
Ordinary Tikhonov regularization consists in minimizing the functional 
2 2 
Fa(2) = ||Kz— yl + ollzll 

for a range of positive regularization parameters a. In iterated Tikhonov regular- 
ization, a > 0 is fixed, an initial approximation x9 is selected (we take x9 = 0 
for simplicity; a general initial approximation requires only small modifications 
to the arguments), and successive approximations are updated by a multistage 


optimization scheme in which the nth approximation is chosen to minimize the 
functional 


F,(2) = ||Kz— yl? tallz— xn, 2 = 1,2,3,... (25) 

This results in the iterative method 
Xn = (K*K +01) '(ax,-1 + K*y), n=1,2,3,.... 
The conventional proof of the convergence of iterated Tikhonov regularization 
uses spectral theory. (See [41] for the more general case of nonstationary iterated 
Tikhonov regularization.) However, the convergence of the method is also an 
immediate consequence of the alternating projection theorem, as we now show. 
Let H be the product Hilbert space H; x H> with norm |-| given by 
(x. y)P = My? + alll’, 

where aq is a fixed positive constant. Note that the graph 


G = {(x, Kx): x € Hy} 


is a closed subspace of H. For a given y € Hp, let 


Ly ={ue H,: Ku= Py}, 
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Fig. 3 The geometry of Ho H 
iterated regularization G 


Py 


Ny 


where P is the orthogonal projector of Hz onto R(K), be the set of least squares 
solutions of Kx = y. One sees that y € D(K‘) if and only if Ly is nonempty. 
If £, = AH, x {Py}, then L, is a closed affine set in the Hilbert space H, and 
y € D(K") & Li NGF g. Furthermore, 


Pe,ng(0,y) = (Kty, Py), 


where Py stands for the metric projector of H onto a closed convex set W C H. 
From the variational characterization (25), one sees that 


Pr, (x0, Kx0) = Pr, (0,0) = (0, Py) 


and Pg(0,Py) = (x1, Kx); therefore (x1,Kx1) = PgPc, (xo, Kx), and 
generally 


(Xn, Kxy) = Fare, (Xn—1, KXn-1) =ao0 = (PgPr,)" (0, y). 


This process of projecting in alternate fashion in the space H is illustrated in Fig. 3. 
The alternating projection theorem then gives 


(Xn, KXn) > Pe,ng0,y) = (KTy, Py), as n—>oo. 
If x are defined as in (25), with y replaced by y’, then it is not difficult to see that 
|x — 2m] < Villy — 9°. 


and hence if || y — y°|| < 5 andn = n(8) — oo as 8 — O, in such as manner that 


/n(5)5 — 0, then xfs) —> Kix. 
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There are many other iterative methods for regularization of ill-posed problems 
(see [19, 49]). Perhaps the simplest is based on the observation that for any A > 0, 
the subspace N(K)* is invariant under the mapping 


F(z) = 1 —AK*K)x + AK*y, 


and K'y is the unique fixed point of F in N(K)+. Furthermore, if 0 < A < 
1/|| K* K'||, then F is a contraction and hence the iterative method 


Xna1 = (I —AK* K) xq + AK*y (26) 


converges to K*y for any x9 € N(K)+. This method was studied for Fredholm 
integral equations of the first kind by Landweber and independently by Fridman. It 
has since become known as Landweber iteration [51]. For this method one can show 
easily that if ||y — y°|| < 6, and x° represents the approximation obtained by (26) 
with y replaced by y®, then xin — Kty,if /n(5)5 > 0. 

The theory of Landweber iteration has been developed for nonlinear operators, 
including a stopping criterion based on the discrepancy principle, by Hanke et al. 
[42] (see also [49]). See [40] for a very useful survey of iterative regularization 
methods. 


Discretization 


> ...numerical precision is the very soul of science ... 
D’Arcy Wentworth Thompson 


The preceding discussion of regularization methods took place in the context 
of general (infinite-dimensional) Hilbert space. However, practical numerical com- 
putations are necessarily finitary. Passing from general elements in an infinite- 
dimensional space to finitely represented approximations involves a process of 
discretization. Discretization of an ill-posed problem can lead to a well-posed 
finite-dimensional problem; however, this discretized version generally will be ill 
conditioned. Regularization methods are meant to address this problem. There 
are two approaches to constructing computable regularizations of linear inverse 
problems; one could handle the ill posedness by first regularizing the infinite- 
dimensional problem and then discretizing the result, or one could discretize the 
original problem and then regularize the resulting ill-conditioned finite-dimensional 
problem. We give a couple of examples of discretized regularizations of the 
former type. (For results on discretized versions of general regularization methods, 
see [34].) 

A key point in the theoretical convergence analysis of regularization methods is 
the interplay between the regularization parameter and the error properties of the 
data. For example, assuming a source condition of the form K'y € R(K*K), the 
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balancing of the rate O(a) for the infinite-dimensional Tikhonov approximation 
Xq using “clean” data with the stability bound 5/./@ for the approximation using 
noisy data leads to the optimal rate O(6?/*) found in (21). To obtain an overall 
convergence rate with respect to the error in data for discretized approximations, 
it is necessary to juggle three balls: a theoretical convergence rate, a measure of 
the quality of the discretization, and a stability bound. In both of the cases we 
consider, the measure of quality of the discretization will be denoted by y,,. In our 
first example, y,, measures how well a given finite-dimensional subspace V, C Ay 
supports the operator K, specifically, 


Ym = ||KU — Pm)|l, 


where P,,, is the orthogonal projector of H, onto V,,. The smaller y,, is, the better the 
subspace V,,, supports the operator K. In the second example, the discretization of a 
regularized version of a Fredholm integral equation of the first kind is accomplished 
by applying a quadrature method to the iterated kernel that generates the operator 
K* K. In this case, y, measures the quality of this quadrature. In both examples it is 
shown that it is theoretically possible to match the optimal rate O(57/) established 
for the infinite-dimensional approximation in (21). 

The variational characterization (22) immediately suggests a Ritz approach 
to discretization, namely, minimization of the Tikhonov functional over a 
finite-dimensional subspace. Note that the global minimum x, of the functional 
F.(-; y) on H, may be characterized by the condition 


(Kxy — Kx, Kv) +a(xq,v) =0, forallv e M, (27) 
where x = Ky. The bilinear form defined on H by 
q(u, v) = (Ku, Kv) + a(u, v) 


is an inner product on Hj, and (27) may be succinctly expressed in terms of this 
inner product as 


q(Xe —xX,v) =O forallv ¢ Ay. 


Suppose that {V,,}°°_, is a sequence of finite-dimensional subspaces of Hy 
satisfying 


VY,CWwWCWwWC...C A and wee Vin = A. 


m=1 


The minimizer xy», of Fy(-; y) over the finite-dimensional subspace V,,, satisfies 


Q(Xum —X;Vm) = 0 forall vm € Vin, 


Linear Inverse Problems 41 
and hence 
q(Xa —Xam;:V¥m) = 9 forall vy, € Vin. 


In other words, Xen = PmXu, where P,, is the projector of H, onto V,,, which is 
orthogonal in the sense of the inner product q(-, -). 
If |-|, denotes the norm on H associated with the inner product q(-, -), that is, 


2 2 2 
Izlg = Kall + alll’, 


then, by the characteristic property of projectors, 
2 2 2 
[Xa Xam|g = |Xa — PinXa q < [Xa — PrnXalg> 


where P,,, is the projector of H onto V,, associated with the (original) inner product 
on #7. But then (since projectors are idempotent), 


|| Xe — Xa,m|l S [Xa — issn le < ||Kxo — KPa eal: 20 = Pa) %ell 
= ||KU— Pay tall + all — Pi Xa? 


= (Ym + a)||(7 _ Patel, 


where 
Ym = ||KU — Pm)|I. 
Therefore, 


I|Xa — Xamll < V1 + ¥m/e || — Pm)Xall. 


If K‘y satisfies the source condition x = K'y € R(K*K), say, x = K* Kw, then 
(I — Pn)Xq¢ = ( — Pm) K*(KK* + a1) KK* Kw, 


and hence 


|Z = Prn)xall < Ym|| Kwll. 
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If Yin = O(a), then we find from (19), 
|KTy — Xa.m|| = O(n). 


In the case of approximate data y° satisfying || — y*|| < 6, one can show, using 
arguments of the same type as above, that a stability bound of the same form as (18) 
holds for the finite-dimensional approximations: 


|| ~~ i) | = 6/Va. 


Taking these results together, we see that if K'y € R(K*K) anda», = @,(64) is 
chosen in sucha way that w,, = C6/3 and y, = O(a), then the finite-dimensional 
approximations achieve the optimal order of convergence: 


| Kty — Xi Gn) | _ 0(8"?). 


Quadrature is another common discretization technique. If a linear inverse 
problem is expressed as a Fredholm integral equation of the first kind 


b 
y(s) = / k(s,t)x(t)dt, c<s<d, 


mapping functions x € L*[a,b] to function y € L?[c,d], then the Tikhonov 
approximation xX, is the solution of the well-posed Fredholm integral equation of 
the second kind 


d db 
/ k(u, s)y(u)du = ax,(s) +f k(s,t)Xe(t)dt, a<s <b, 


where the iterated kernel k (-, -) is given by 


d 
k(s,t) =| ktu,s)k(u,t)du, a<s,t <b. 
If a convergent quadrature scheme with positive weights wen , and nodes 
j=l 


m 
{ui } , 1s applied to the iterated kernel, a degenerate kernel 
j=l 


m 


km(s, th= » wk a” s) k (a r) 


j=l 
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results, converting the infinite-dimensional Tikhonov problem into the finite rank 
problem 


AXeam + ae = K*y (28) 


where 


Rnz= Yow {k;,ky and kj(s) =k (us), 


j=l 


The problem (28) is equivalent to an m x m linear algebraic system with a unique 
solution. The convergence of the approximations resulting from this finite system 
to the infinite-dimensional Tikhonov approximation x, € L*[a,b] depends on the 
number 


Yn = \|Kmn — K*K|. 


Ifa = a(m) > Oasm > ow and yy, = O(a(m)), then it can be shown that 
Xa(m),m > K t y. Furthermore, a stability bound of the form O(6/./a) holds under 
appropriate conditions, and one can show that the optimal rate O(57/) is achievable 
if the parameters governing the finite-dimensional approximations are appropriately 
related [29]. A much more extensive analysis along these lines is carried out in [11]. 
For more on numerical methods for discrete inverse problems, see [43, 70]. 


5 Conclusion 


> Eventually, we reach the dim boundary ... 
There, we measure shadows... 
Edwin Hubble 


The first book devoted exclusively to the mathematical theory of inverse and 
ill-posed problems was that of Tikhonov and Arsenin [68]. Kirsch [50] is a fine 
treatment of the general theory of inverse problems, and Eng] et al. [19] is the best 
comprehensive presentation of the theory of regularization for inverse and ill-posed 
problems. Other useful books on the general topic are [46] and [69]. A number of 
books and survey articles treat inverse theory in a specific context. Some of the 
areas treated include astronomy [16], engineering [45], geophysics [58], imaging 
[6,7,9, 10,20,44,55,62], mathematical physics [24], oceanography [4,73], parameter 
estimation [3], indirect measurement [2], and vibration analysis [23]. 
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Abstract 


Large-scale inverse problems arise in a variety of significant applications in 
image processing, and efficient regularization methods are needed to compute 
meaningful solutions. This chapter surveys three common mathematical models 
including a linear model, a separable nonlinear model, and a general nonlinear 
model. Techniques for regularization and large-scale implementations are con- 
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sidered, with particular focus on algorithms and computations that can exploit 
structure in the problem. Examples from image deconvolution, multi-frame blind 
deconvolution, and tomosynthesis illustrate the potential of these algorithms. 
Much progress has been made in the field of large-scale inverse problems, but 
many challenges still remain for future research. 


1 Introduction 


Powerful imaging technologies, including very large telescopes, synthetic aperture 
radar, medical imaging scanners, and modern microscopes, typically combine a 
device that collects electromagnetic energy (e.g., photons) with a computer that 
assembles the collected data into images that can be viewed by practitioners, such 
as scientists and doctors. The “assembling” process typically involves solving an 
inverse problem; that is, the image is reconstructed from indirect measurements of 
the corresponding object. Many inverse problems are also ill-posed, meaning that 
small changes in the measured data can lead to large changes in the solution, and 
special tools or techniques are needed to deal with this instability. In fact, because 
real data will not be exact (it will contain at least some small amount of noise or 
other errors from the data collection device), it is not possible to find the exact 
solution. Instead, a physically realistic approximation is sought. This is done by 
formulating an appropriate regularized (i.e., stabilized) problem, from which a good 
approximate solution can be computed. 

Inverse problems are ubiquitous in imaging applications, including deconvo- 
lution (or, more generally, deblurring) [1,51], superresolution (or image fusion) 
[18, 27], image registration [70], image reconstruction [74, 75], seismic imaging 
[31], inverse scattering [15], and radar imaging [17]. These problems are referred to 
as large-scale because they typically require processing a large amount of data (the 
number of pixels or voxels in the discretized image) and systems with a large (e.g., 
10° for a 3D image reconstruction problem) number of equations. Mathematicians 
began to rigorously study inverse problems in the 1960s, and this interest has 
continued to grow over the past few decades due to applications in fields such as 
biomedical, seismic, and radar imaging; see, for example, [12,28,47,49, 99] and the 
references therein. 

We remark that the discussion in this chapter does not address some very 
important issues that can arise in PDE-based inverse problems, such as adjoints 
and proper meshing. Inverse problems such as these arise in important applications, 
including PDE parameter identification, seismic imaging, and inverse scattering; we 
refer those interested in these topics and applications to the associated chapters in 
this handbook and the references therein. 

This chapter discusses computational approaches to compute approximate solu- 
tions of large-scale inverse problems. Mathematical models and some applications 
are presented in Sect.2. Three basic models are considered: a general nonlinear 
model, a linear model, and a mixed linear/nonlinear model. Several regularization 
approaches are described in Sect. 3. Numerical methods that can be used to compute 
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approximate solutions for the three basic models, along with illustrative examples 
from specific imaging applications, are described in Sect. 4. Concluding remarks, 
including a partial list of open questions, are provided in Sect. 5. 


2 Background 


A mathematical framework for inverse problems is presented in this chapter, 
including model problems and imaging applications. Although only a limited 
number of imaging applications are considered, the model problems, which range 
from linear to nonlinear, are fairly general and can be used to describe many other 
applications. For more complete treatments of inverse problems and regularization, 
see [12, 28, 47, 49, 50,99]. 


Model Problems 


An inverse problem involves the estimation of certain quantities using information 
obtained from indirect measurements. A general mathematical model to describe 
this process is given by 


Dexact = F (Xexact), (1) 


where Xexact denotes the exact (or ideal) quantities that need to be estimated and Dexact 
is used to represent perfectly measured (error-free) data. The function F is defined 
by the data collection process and is assumed known. Typically, it is assumed that 
F is defined on Hilbert spaces and that it is continuous and weakly sequentially 
closed [29]. 

Unfortunately, in any real application, it is impossible to collect error-free data, 
so a more realistic model of the data collection process is given by 


b= F (Xexact) ai UB (2) 


where y represents noise and other errors in the measured data. The precise form of 
F depends on the application; the following three general problems are considered 
in this chapter: 


¢ For linear problems, F(x) = Ax, where A is a linear operator. In this case, the 
data collection process is modeled as 


b= AXexact Si n, 
and the inverse problem is: given b and A, compute an approximation of Xexact, 


¢ In some cases, x can be separated into two distinct components, x and x”, 
with F(x) = F (x©,x9) = A (x9) x, where A is a linear operator defined 
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by x“, That is, the data b depends linearly on x and nonlinearly on x”. In 
this case, the data collection process is modeled as 


(ne (e 
b — A (xe) < + n, 


and the inverse problem is: given b and the parametric form of A, compute 
approximations of x) and xO 
e If the problem is not linear or separable, as described above, then the general 


nonlinear model, 
b= F (Xexact) oh n, 


will be considered. In this case, the inverse problem is: given b and F’, compute 
an approximation of Xexact. 


In most of what follows, it is assumed that the problem has been discretized, so x, b, 
and 7 are vectors, and A is a matrix. Depending on the constraints assumed and the 
complexity of the model used, problems may range from linear to fully nonlinear. 
This is true of the applications described in the next subsection. 


Imaging Applications 


Three applications in image processing that lead to inverse problems are discussed 
in this subsection. For each application, the underlying mathematical model is 
described, and some background for the problem is presented. The formulation of 
each of these problems results in linear, separable, and nonlinear inverse problems, 
respectively. 


Image Deblurring and Deconvolution 

In many important applications, such as when ground-based telescopes are used to 
observe objects in space, the observed image is degraded by blurring and noise. 
Although the blurring can be partially avoided by using sophisticated and expensive 
imaging devices, computational post-processing techniques are also often needed to 
further improve the resolution of the image. This post-processing is known as image 
deblurring. To give a precise mathematical model of image deblurring, suppose 
x(t),t € R4, is a scalar function describing the true d-dimensional (e.g., for a 
plane image containing pixels, d = 2) image. Then the observed, blurred, and noisy 
image is given by 


b(s) = [ konxiat +16) (3) 
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where s € R@ and n(s) represents additive noise. The kernel k(s, f) is a function that 
specifies how the points in the image are distorted and is therefore called the point 
spread function (PSF). The inverse problem of image deblurring is: given k and b, 
compute an approximation of x. If the kernel has the property that k(s,t) = k(s—t), 
then the PSF is said to be spatially invariant; otherwise, it is said to be spatially 
variant. In the spatially invariant case, the blurring operation, f k(s — t)x(t)dt, is 
convolution, and thus the corresponding inverse problem is called deconvolution. 

In a realistic problem, images are collected only at discrete points (pixels or 
voxels) and are only available in a finite bounded region. Therefore, one must 
usually work directly either with a semi-discrete model 


b(s;) = f koj.NxOdt +n j=il,...,N 
Q 


where JN is the number of pixels or voxels in the observed image or with the fully 
discrete model 


b= AXexact - 7, 


where Xexact, b, and 4 are vectors obtained by discretizing functions x, b, and 7 
and A is a matrix that arises when approximating the integration operation with, 
for example, a quadrature rule. Moreover, a precise kernel representation of the 
PSF may not be known but instead must be constructed experimentally from the 
imaging system by generating images of “point sources.” What constitutes a point 
source depends on the application. For example, in atmospheric imaging, the point 
source can be a single bright star [53]. In microscopy, the point source is typically a 
fluorescent microsphere having a diameter that is about half the diffraction limit of 
the lens [24]. For general motion blurs, the PSF is described by the direction (e.g., 
angle) and speed at which objects are moving [56]. 

For spatially invariant blurs, one point source image and appropriate boundary 
conditions are enough to describe the matrix A. This situation has been well studied; 
algorithms to compute approximations of x can be implemented efficiently with fast 
Fourier transforms (FFT) or other trigonometric transforms [1,51]. More recently, 
an approach has been proposed where the data can be transformed to the Radon 
domain so that computations can be done efficiently with, for example, wavelet 
filtering techniques [26]. 

Spatially variant blurs also occur in a variety of important applications. For 
example, in positron emission tomography (PET), patient motion during the 
relatively long scan times causes reconstructed images to be corrupted by nonlinear, 
nonuniform spatially variant motion blur [33, 84]. Spatially variant blurs also occur 
when the object and image coordinates are tilted relative to each other, as well 
as in X-ray projection imaging [100], lens distortions [65], and wave aberrations 
[65]. Moreover, it is unlikely that the blur is truly spatially invariant in any realistic 
application, especially over large image planes. 
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Various techniques have been proposed to approximately model spatially variant 
blurs. For example, in the case of patient motion in PET brain imaging, a motion 
detection device is used to monitor the position of the patient’s head during the 
scan time. This information can then be used to construct a large sparse matrix 
A that models the motion blur. Other, more general techniques include coordinate 
transformation [68], image partitioning [93], and PSF interpolation [72, 73]. 


Multi-Frame Blind Deconvolution 

The image deblurring problem described in the previous subsection assumes that the 
blurring operator, or PSF, is known. However, in most cases, only an approximation 
of the operator, or an approximation of parameters that defines the operator, 
is known. For example, as previously mentioned, the PSF is often constructed 
experimentally from the imaging system by generating images of point sources. 
In many cases, such approximations are fairly good and are used to construct the 
matrix A in the linear model. However, there are situations where it is not possible 
to obtain good approximations of the blurring operator, and it is necessary to include 
this knowledge in the mathematical model. Specifically, consider the general image 
formation model 


b=A (x2) <5 +H (4) 


where b is a vector representing the observed, blurred, and noisy image and x 


is a vector representing the unknown true image to be reconstructed. A (xe) is 


an ill-conditioned matrix defining the blurring operator. For example, in the case of 
spatially invariant blurs, en could simply be the pixel (image space) values of the 
PSF. Or xr) could be a small set of parameters that define the PSF, such as with a 
Zernike polynomial-based representation [67]. In general, the number of parameters 
defining x) is significantly smaller than the number of pixels in the observed 
image. As in the previous subsection, 7 is a vector that represents unknown additive 
noise in the measured data. The term blind deconvolution is used for algorithms that 
attempt to jointly compute approximations of x and xo from the separable 
inverse problem given by Eq. (4). 

Blind deconvolution problems are highly underdetermined, which present many 
challenges to optimization algorithms that can easily become trapped in local 
minima. This difficulty has been well-documented; see, for example, [64, 67]. To 
address challenges of nonuniqueness, it may be necessary to include additional 
constraints, such as nonnegativity and bounds on the computed approximations x”) 
and x, 

Multi-frame blind deconvolution (MFBD) [64, 67] reduces some of the 
nonuniqueness problems by collecting multiple images of the same object, but 
with different blurring operators. Specifically, suppose a set of (e.g., mm) observed 


images of the same object are modeled as 


b; = A(x) a+ a, i =1,2,...,m. (5) 
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Then, a general separable inverse problem of the form given by Eq. (4) can be 
obtained by setting 


£ 
b; x Ua 
£) : 
b= ’ . = : ’ y= 
Dn x ” Nin 


Although multiple frames reduce, to some extent, the nonuniqueness problem, 
they do not completely eliminate it. In addition, compared to single-frame blind 
deconvolution, there is a significant increase in the computational complexity of 
processing the large, multiple data sets. 

There are many approaches to solving the blind and multi-frame blind deconvo- 
lution problem; see, for example, [14]. In addition, many other imaging applications 
require solving separable inverse problems, including super resolution (which is an 
example of image data fusion) [18, 27,57, 76], the reconstruction of 3D macro- 
molecular structures from 2D electron microscopy images of cryogenically frozen 
samples (Cryo-EM) [22, 35,55, 66, 82, 88], and seismic imaging applications [40]. 


Tomosynthesis 

Modern conventional X-ray systems that use digital technology have many benefits 
to the classical film X-ray systems, including the ability to obtain high-quality 
images with lower-dosage X-rays. The term “conventional” is used to refer to a 
system that produces a 2D projection image of a 3D object, as opposed to computed 
tomography (CT), which produces 3D images. Because of the inexpensive cost, low 
X-ray dosage, and ease of use, digital X-ray systems are widely used in medicine, 
from emergency rooms, to mammography, to dentistry. 

Tomosynthesis is a technique that can produce 3D image information of an object 
using conventional X-ray systems [25]. The basic idea underlying tomosynthesis 
is that multiple 2D image projections of the object are taken at varying incident 
angles, and each 2D image provides different information about the 3D object. See 
Fig. | for an illustration of a typical geometry for breast tomosynthesis imaging. 
The relationship between the multiple 2D image projections and the 3D object can 
be modeled as a nonlinear inverse problem. Reconstruction algorithms that solve 
this inverse problem should be able to reconstruct any number of slices of the 3D 
object. Sophisticated approaches used for 3D CT reconstruction cannot be applied 
here because projections are only taken from a limited angular range, leaving entire 
regions of the frequency space unsampled. Thus, alternative approaches need to be 
considered. 

The mathematical model described in this section is specifically designed for 
breast imaging and assumes a polyenergetic (i.e., multiple energy) X-ray source. 
It is first necessary to determine what quantity will be reconstructed. Although 
most X-ray projection models are derived in terms of the values of the attenuation 
coefficients for the voxels, it is common in breast imaging to interpret the voxels 
as a composition of adipose tissue, glandular tissue, or a combination of both [42]. 
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Fig. 1 Breast tomosynthesis example. Typical geometry of the imaging device used in breast 
imaging 


Thus, each voxel of the object can be represented using the percentage glandular 
fraction, that is, the percentage of glandular tissue present in that voxel. If density 
or attenuation coefficient values are desired, then these can be obtained from the 
glandular fraction through a simple algebraic transformation. 

Now assume that the 3D object is discretized into a regular grid of voxels and 
that each of the 2D projection images is discretized into a regular grid of pixels. 
Specifically, let N represent the number of voxels in the discretized 3D object and 
let M be the number of pixels in a discretized 2D projection image. In practice, N 
is on the order of a few billion and / is the order of a few million, depending on 
the size of the imaging detector. The energy-dependent linear attenuation coefficient 
for voxel 7 = 1,2,..., N in the breast can be represented as 


we) = s(e)xetder + 2), 
where x, represents the percentage glandular fraction in voxel j of the “true” 
object and s(e) and z(e) are known energy-dependent linear fit coefficients. This 
type of decomposition to reduce the number of degrees of freedom, which is 
described in more detail in [20], is similar to an approach used by De Man et al. [23] 
for CT, in which they express the energy-dependent linear attenuation coefficient in 
terms of its photoelectric component and Compton scatter component. 

The projections are taken from various angles in a predetermined angular range, 
and the photon energies can be discretized into a fixed number of levels. Let there 
be mg angular projections and assume the incident X-ray has been discretized into 
Ne photon energy levels. In practice, a typical scan may have ng = 21 andn, = 43. 
For a particular projection angle, compute a monochromatic ray trace for one energy 


Large-Scale Inverse Problems in Imaging 55 


level and then sum over all energies. Let a“) represent the length of the ray that 
passes through voxel 7, contributing to pixel 7. Then, the discrete monochromatic 
ray trace for pixel i can be represented by 


N N N 
Di we)\Pa) = s(e) DT xed + ele) Da. (6) 
j=l 


j=l j=l 


Using the standard mathematical model for transmission radiography, the ith pixel 
value for the @th noise-free projection image, incorporating all photon energies 
present in the incident X-ray spectrum, can be written as 


Ne 


N 
bs’ = > a(evexp|— > w@)Ma® |, (7) 
e=1 


j=l 


where gQ(e) is a product of the current energy with the number of incident photons 
at that energy. To simplify notation, define Ag to be an M x N matrix with entries 
a“), Then Eq. (6) gives the ith entry of the vector 


S(€)AgXexact + z(e)Aoel, 


where Xexact is a vector whose jth entry is 52, and 1 is a vector of all ones. 
Furthermore, the 6th noise-free projection image in vector form can be written as 


Ne 


be = >> o(e) exp(—[s(e)AgXexact + <(e)Adl]), (8) 


e=1 


where the exponential function is applied component-wise. 

Tomosynthesis reconstruction is a nonlinear inverse problem where the goal is 
to approximate the volume, Xexact, given the set of projection images from various 
angles, bg, 9 = 1,2,....,n9. This can be put in the general nonlinear model 


b= F (Xexact) + UB 


where 


. X ole) exp(—[s(e)Arx + s(@)A1) 


b= : and F(x) = 


Dow 5* o(e) exp(—[s(e)AngX + 2(€) Ang 11) 


e=1 
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3 Mathematical Modeling and Analysis 


A significant challenge when attempting to compute approximate solutions of 
inverse problems is that they are typically ill-posed. To be precise, in 1902 
Hadamard defined a well-posed problem as one that satisfies the following require- 
ments: 


1. The solution is unique; 
2. The solution exists for arbitrary data; 
3. The solution depends continuously on the data. 


Ill-posed problems, and hence most inverse problems, typically fail to satisfy at 
least one of these criteria. It is worth mentioning that this definition of an ill- 
posed problem applies to continuous mathematical models and not precisely to the 
discrete approximations used in computational methods. However, the properties 
of the continuous ill-posed problem are often carried over to the discrete problem 
in the form of a particular kind of ill-conditioning, making certain (usually high 
frequency) components of the solution very sensitive to errors in the measured data; 
this property is discussed in more detail for linear problems in section “Linear 
Problems.” Of course, this may depend on the level of discretization; a coarsely 
discretized problem may not be very ill-conditioned, but it also may not bear much 
similarity to the underlying continuous problem. 

Regularization is a term used to refer to various techniques that modify the 
inverse problem in an attempt to overcome the instability caused by ill-posedness. 
Regularization seeks to incorporate a priori knowledge into the solution process. 
Such knowledge may include information about the amount or type of noise, the 
smoothness or sparsity of the solution, or restrictions on the values the solution 
may obtain. Each regularization method also requires choosing one or more 
regularization parameters. A variety of approaches are discussed in this section. 

The theory for regularizing linear problems is much more developed than it is 
for nonlinear problems. This is due, in large part, to the fact that the numerical 
treatment of nonlinear inverse problems is often highly dependent on the particular 
application. However, good intuition can be gained by first studying linear inverse 
problems. 


Linear Problems 


Consider the linear inverse problem 
b= AXexact + n, 


where b and A are known, and the aim is to compute an approximation of Xexact. The 
linear problem is a good place to illustrate the challenges that arise when attempting 
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to solve large-scale inverse problems. In addition, some of the regularization 
methods and iterative algorithms discussed here can be used in, or generalized for, 
nonlinear inverse problems. 


SVD Analysis 
A useful tool in studying linear inverse problems is the singular value decomposition 
(SVD). Any m x n matrix A can be written as 


A=UZV" (9) 


where U is an m xm orthogonal matrix, V is an xn orthogonal matrix, and & is an 
m Xn diagonal matrix containing the singular values 0; > 02 > ... = Omin(m.n) = 0. 
If A is nonsingular, then an approximation of Xexact is given by the inverse solution 


n T n 7 n 7 
—!] u; b u; Dexact u; U] 
Xiny = A b=) v=) vt) Vi 
‘ on : Oj Oj 
i=1 i=1 ——-— i=1 ——— 
Xexact error 


where u,; and v; are the singular vectors of A (i.e., the columns of U and 
V, respectively). As indicated above, the inverse solution is comprised of two 
components: Xexact and an error term. Before discussing algorithms to compute 
approximations Of Xexact, it is useful to study the error term. 

For matrices arising from ill-posed inverse problems, the following properties 
hold: 


¢ The matrix A is severely ill-conditioned, with the singular values o; decaying to 
zero without a significant gap to indicate numerical rank. 

¢ The singular vectors corresponding to the small singular values tend to oscillate 
more (i.e., have higher frequency) than singular vectors corresponding to large 
singular values. 

e The components ju Dexact| decay on average faster than the singular values o;. 
This is referred to as the discrete Picard condition [49]. 


The first two properties imply that the high-frequency components of the error term 
are highly magnified by division of small singular values. The computed inverse 
solution is dominated by these high-frequency components and is in general a 
very poor approximation of Xexact. However, the third property suggests that there 
is hope of reconstructing some information about Xexact; that is, an approximate 
solution can be obtained by reconstructing components corresponding to the large 
singular values and filtering out components corresponding to small singular 
values. 
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Regularization by SVD Filtering 

The SVD filtering approach to regularization is motivated by observations made 
in the previous subsection. That is, by filtering out components of the solution 
corresponding to the small singular values, a reasonable approximation of Xexac, Can 
be computed. Specifically, an SVD-filtered solution is given by 


n F 

u; b 

Xait = i u Vi, 10 

im d bi (10) 

where the filter factors, ¢;, satisfy ¢; ~ 1 for large o;, and ¢; ~ O for small o;. 
That is, the large singular value components of the solution are reconstructed, 

while the components corresponding to the small singular values are filtered out. 

Different choices of filter factors lead to different methods. Some examples include: 


Truncated SVD filter Tikhonov filter Exponential filter 


1 if Oo; >T o2 2,2 
— a = | — eG /e 
# 0 if o<t a of + a ; 


Note that using a Taylor series expansion of the exponential term in the exponential 
filter, it is not difficult to see that the Tikhonov filter is a truncated approximation 
of the exponential filter. Moreover, the Tikhonov filter has an equivalent variational 
form, which is described in section “Variational Regularization and Constraints”. 

Observe that each of the filtering methods has a parameter (e.g., in the above 
examples, t and a) that needs to be chosen to specify how much filtering is done. 
Appropriate values depend on properties of the matrix A (i.e., on its singular 
values and singular vectors) as well as on the data, b. Some techniques to help 
guide the choice of the regularization parameter are discussed in section “Choosing 
Regularization Parameters.” 

Because the SVD can be very expensive to compute for large matrices, this 
explicit filtering approach is generally not used for large-scale inverse problems. 
There are some exceptions, though, if A is highly structured. For example, suppose 
A can be decomposed as a Kronecker product, 


aA ak tee aA. 
aa, aA, -- aA, 


aa. aA, see aa, 


where A, is an m Xx m matrix, and A; is ann x n matrix with entries denoted by 
a Then this block structure can be exploited when computing the SVD and when 
implementing filtering algorithms [51]. 
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It is also sometimes possible to use an alternative factorization. Specifically, 
suppose that 


A= QAQ*, 


where A is a diagonal matrix and Q* is the complex conjugate transpose of Q, 
with Q*Q = I. This is called a spectral factorization, where the columns of Q are 
eigenvectors and the diagonal elements of A are the eigenvalues of A. Although 
every matrix has an SVD, only normal matrices (i.e., matrices that satisfy A*A = 
AA“) have a spectral decomposition. However, if A has a spectral factorization, then 
it can be used, in place of the SVD, to implement the filtering methods described in 
this section. The advantage is that it is sometimes more computationally convenient 
to compute a spectral decomposition than an SVD; an example of this is given in 
section “Linear Example: Deconvolution.” 


Variational Regularization and Constraints 
Variational regularization methods have the form 


min {Ib — Axl; + a 7(x)} , (11) 


where the regularization operator 7 and the regularization parameter a must be 
chosen. The variational form provides a lot of flexibility. For example, one could 
include additional constraints on the solution, such as nonnegativity, or it may be 
preferable to replace the least squares criterion with the Poisson log likelihood 
function [3-5]. As with filtering, there are many choices for the regularization 
operator, 7, such as Tikhonov, total variation [16, 85,99], and sparsity constraints 
[13,34, 94]: 


Tikhonov Total variation Sparsity 


I(x) =|Lx3 J) =| VOX FOR] 700 = [loath 


Tikhonov regularization, which was first proposed and studied extensively in the 
early 1960s [69,83,89-—91], is perhaps the most well-known approach to regularizing 
ill-posed problems. L is typically chosen to be the identity matrix, or a discrete 
approximation to a derivative operator, such as the Laplacian. If L = I, then it is 
not difficult to show that the resulting variational form of Tikhonov regularization, 
namely, 


min {\|b— Axl} + a [IxI3} (12) 


can be written in an equivalent filtering framework by replacing A with its SVD 
[49]. 
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For total variation, D;, and D,, denote discrete approximations of horizontal and 
vertical derivatives of the 2D image x, and the approach extends to 3D images in an 
obvious way. Efficient and stable implementation of total variation regularization is 
a nontrivial problem; see [16,99] and the references therein for further details. 

In the case of sparse reconstructions, the matrix ® represents a basis in which the 
image, x, is sparse. For example, for astronomical images that contain a few bright 
objects surrounded by a significant amount of black background, an appropriate 
choice for ® might be the identity matrix. Clearly, the choice of ® is highly 
dependent on the structure of the image x. The usage of sparsity constraints for 
regularization is currently a very active field of research, with many open problems. 
We refer interested readers to the chapter in this handbook on compressive sensing, 
and the references therein. 

We also mention that when the majority of the elements in the image x are 
zero or near zero, as may be the case for astronomical or medical images, it 
may be wise to enforce nonnegativity constraints on the solution [4,5, 99]. This 
requires that each element of the computed solution x is not negative, which is 
often written as x > 0. Though these constraints add a level of difficulty when 
solving, they can produce results that are more feasible than when nonnegativity is 
ignored. 

Finally, it should be noted that depending on the structure of matrix A, the type of 
regularization, and the additional constraints, a variety of optimization algorithms 
can be used to solve (11). In some cases, it is possible to use a very efficient filtering 
approach, but typically it is necessary to use an iterative method. 


Iterative Regularization 

As mentioned in section “Variational Regularization and Constraints,” iterative 
methods are often needed to solve the variational form of the regularized problem. 
An alternate approach to using variational regularization is to simply apply the 
iterative method to the least squares problem, 


min ||b — Ax([5 . 
x 


Note that if an iterative method applied to this unregularized problem is allowed 
to “converge,” it will converge to an inverse solution, Xiny, which is corrupted by 
noise (recall the discussion in section “SVD Analysis”). However, many iterative 
methods have the property (provided the problem on which it is applied satisfies 
the discrete Picard condition) that the early iterations reconstruct components of the 
solution corresponding to large singular values, while components corresponding 
to small singular values are reconstructed at later iterations. Thus, there is an 
observed “semi-convergence” behavior in the quality of the reconstruction, whereby 
the approximate solution improves at early iterations and then degrades at later 
iterations (a more detailed discussion of this behavior is given in section “Hybrid 
Iterative-Direct Regularization” in the context of the iterative method LSQR). If 
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the iteration is terminated at an appropriate point, a regularized approximation 
of the solution is computed. Thus, the iteration index acts as the regularization 
parameter, and the associated scheme is referred to as an iterative regularization 
method. 

Many algorithms can be used as iterative regularization methods, including 
Landweber [61], steepest descent, and the conjugate gradient method (e.g., for 
nonsymmetric problems, the CGLS implementation [8] or the LSQR implemen- 
tation [80, 81], and for symmetric indefinite problems, the MR-II implementation 
[43]). Most iterative regularization methods can be put into a general framework 
associated with solving the minimization problem 


1 
min f(x) = 5x AT AK —x'A'b (13) 
with a general iterative method of the form 
X41 = Xk + pxMy (Ab — ATAX,) = Xx + pxMare, (14) 


where r;, = A’ b — A’ Ax,. With specific choices of p, and M;, one can obtain a 
variety of well-known iterative methods: 


¢ The Landweber method is obtained by taking py = p (i.e., 9 remains constant 
for each iteration) and M; = I (the identity matrix). Due to its very slow 
convergence, this classic approach is not often used for linear inverse problems. 
However, it is very easy to analyze the regularization properties of the Landweber 
iteration, and it can be useful for certain large-scale nonlinear ill-posed inverse 
problems. 

¢ The steepest descent method is produced if M; = I is again fixed as the identity, 
but now p; is chosen to minimize the residual at each iteration. That is, px, is 
chosen as 


Pr = argmin f(x; + prix). 
p>0 


e Again, this method typically has very slow convergence, but with proper 
preconditioning it may be competitive with other methods. 
¢ It is also possible to obtain the conjugate gradient method by setting Mo = I 


Tr 
and My4; = I- Te , where sp = Xg41 — Xx and yx = ATA(KK41 — x;). As 
k Ok 
with the steepest descent method, p; is chosen to minimize the residual at each 
iteration. Generally, the conjugate gradient method converges much more quickly 


than Landweber or steepest descent. 


Other iterative algorithms that can be put into this general frame work include 
the Brakhage v methods [10] and Barzilai and Borwein’s lagged steepest descent 
scheme [6]. 
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Hybrid Iterative-Direct Regularization 

One of the main disadvantages of iterative regularization methods is that it 
can be very difficult to determine appropriate stopping criteria. To address this 
problem, work has been done to develop hybrid methods that combine variational 
approaches with iterative methods. That is, an iterative method, such as the LSQR 
implementation of the conjugate gradient method, is applied to the least squares 
problem min ||Ax — b| Bi and variational regularization is incorporated within the 
iteration process. To understand how this can be done, it is necessary to briefly 
describe how the LSQR iterates are computed. 

LSQR is based on the Golub-Kahan (sometimes referred to as Lanczos) bidiag- 
onalization (GKB) process. Given an m x n matrix A and vector b, the kth GKB 
iteration computes an m x (k + 1) matrix W;, ann x k matrix Y;, an x 1 vector 
Yx+1, anda (k + 1) x k bidiagonal matrix B, such that 


ATW = YiBe + Vet 1Ye+1eK 41 (15) 
AY; = W;By, (16) 
where e;, denotes the (k + 1)st standard unit vector and B,; has the form 


Y1 
Bo y2 


B= (17) 


Be Vk 
Bri 


Matrices W, and Y; have orthonormal columns, and the first column of W, is 
b/||b||2. Given these relations, an approximate solution x; can be computed from 
the projected least squares problem 


in ||Ax — bl]; = min ||B,X — Ber ||; 18 
amin, Ax bl; = min |Be& — Be: [3 (18) 
where 8 = ||b||2 and x, = Y;x. An efficient implementation of LSQR does not 


require storing the matrices W, and Y; and uses an efficient updating scheme to 
compute xX at each iteration; see [81] for details. 

An important property of GKB is that for small values of k, the singular values 
of the matrix By, approximate very well certain singular values of A, with the 
quality of the approximation depending on the relative spread of the singular values; 
specifically, the larger the relative spread, the better the approximation [8, 37, 87]. 
For ill-posed inverse problems, the singular values decay to and cluster at zero, such 
as o; = O(i~°) where c > 1 oro; = O(c’) where 0 < c < 1 andi = 1,2,...,n 
[95, 96]. Thus, the relative gap between large singular values is generally much 
larger than the relative gap between small singular values. Therefore, if the GKB 
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Fig. 2 This figure shows plots of the singular values of A, denoted as o;(A) (left plot), and the 
relative spread of A’s singular values (right plot) 


iteration is applied to a linear system arising from discretization of an ill-posed 
inverse problem, then the singular values of By; converge very quickly to the largest 
singular values of A. The following example illustrates this situation. 


Example I. Consider a linear system obtained by discretization of a one- 
dimensional first-kind Fredholm integral equation of the form (3), where the kernel 
k(s,t) is given by the Green’s function for the second derivative and which is 
constructed using deriv2 in the MATLAB package Regularization Tools [48]. 
Although this is not an imaging example, it is a small-scale canonical ill-posed 
inverse problem that has properties found in imaging applications. The deriv2 
function constructs an n x n matrix A from the kernel 


s(t—1) ifs <t 

k(s,t) = 
(s,1) t(s—1) ifs>t 

defined on [0, 1] x [0, 1]. We use » = 256. There are also several choices for 

constructing vectors Xexact aNd Dexact (see [48]), but we focus only on the matrix A 

in this example. 


Figure 2 shows a plot of the singular values of A and their relative spread; that is, 


0; (A) — 0;41(A) 
0; (A) 
where the notation 0;(A) is used to denote the ith largest singular value of A. 
Figure 2 clearly illustrates the properties of ill-posed inverse problems; the singular 
values of A decay to and cluster at 0. Moreover, it can be observed that in general the 
relative gap of the singular values is larger for the large singular values and smaller 
for the smaller singular values. Thus, for small values of k, the singular values of 
By, converge quickly to the large singular values of A. This can be seen in Fig. 3, 
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Fig. 3 The plots in the /eft column of this figure show the singular values of A, denoted as 0; (A), 


along with the singular values of B;, denoted as 0; (Bx), for k = 10, 20, and 50. The plots in the 
a; (A)—a; (Bx) | 


right column show the relative difference, - 
oi (A) 


which compare the singular values of A with those of the bidiagonal matrix B; for 
k = 10, 20, and 50. 

This example implies that if LSQR is applied to the least squares problem 
min ||Ax — b||2, then at early iterations the approximate solutions x; will be in a 


subspace that approximates a subspace spanned by the large singular components 
of A. Thus, for k < n,x, is a regularized solution. However, eventually x, 
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should converge to the inverse solution, which is corrupted with noise (recall the 
discussion in section “Iterative Regularization’’). This means that the iteration index 
k plays the role of a regularization parameter; if k is too small, then the computed 
approximation x; is an over-smoothed solution, while if k is too large, x, is 
corrupted with noise. Again, we emphasize that this semi-convergence behavior 
requires that the problem satisfies the discrete Picard condition. More extensive 
theoretical arguments of this semi-convergence behavior of conjugate gradient 
methods can be found elsewhere; see [43] and the references therein. 

Instead of early termination of the iteration, hybrid approaches enforce regular- 
ization at each iteration of the GKB method. Hybrid methods were first proposed by 
O’Leary and Simmons in 1981 [78] and later by Bjorck in 1988 [7]. The basic idea 
is to regularize the projected least squares problem (18) involving B;, which can 
be done very cheaply because of the smaller size of B;,. More specifically, because 
the singular values of B; approximate those of A, as the GKB iteration proceeds, 
the matrix B, becomes more ill-conditioned. The iteration can be stabilized by 
including Tikhonov regularization in the projected least square problem (18) to 
obtain 


min {|[B.&— Bei||} + a? |} (19) 


where again B = ||b||. and x, = Y,;x. Thus, at each iteration it is necessary to 
solve a regularized least squares problem involving a bidiagonal matrix B;. Notice 
that since the dimension of B;, is very small compared to A, it is much easier to 
solve for X in Eq. (19) than it is to solve for x in the full Tikhonov regularized 
problem (12). More importantly, when solving Eq. (19) one can use sophisticated 
parameter choice methods to find a suitable @ at each iteration. 

To summarize, hybrid methods have the following benefits: 


¢ Powerful regularization parameter choice methods can be implemented effi- 
ciently on the projected problem. 

¢ Semi-convergence behavior of the relative errors observed in LSQR is avoided, 
sO an imprecise (over) estimate of the stopping iteration does not have a 
deleterious effect on the computed solution. 


Realizing these benefits in practice, though, is nontrivial. Thus, various authors have 
considered computational and implementation issues, such as robust approaches 
to choose regularization parameters and stopping iterations; see, for example, 
[9, 11, 21, 45, 60, 62, 78]. We also remark that our discussion of hybrid methods 
focused on the case of Tikhonov regularization with L = I. Implementation of 
hybrid methods when L is not the identity matrix, such as a differentiation operator, 
can be nontrivial; see, for example, [50,59]. 
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Choosing Regularization Parameters 
Each of the regularization methods discussed in this section requires choosing a 
regularization parameter. It is a nontrivial matter to choose “optimal” regularization 
parameters, but there are methods that can be used as guides. Some require a priori 
information, such as a bound on the noise or a bound on the solution. Others attempt 
to estimate an appropriate regularization parameter directly from the given data. 

To describe some of the more popular parameter choice methods, let x,-. denote 
a solution computed by a particular regularization method: 


¢ Discrepancy Principle. In this approach, a solution is sought such that 


|b — Axreg 


2= TMM 


Where t > | is a predetermined number [71]. This is perhaps the easiest of the 
methods to implement, and there are substantial theoretical results establishing 
its behavior in the presence of noise. However, it is necessary to have a good 
estimate for ||7||2. 

¢ Generalized Cross Validation. The idea behind generalized cross validation 
(GCV) is that if one data point is removed from the problem, then a good 
regularized solution should predict that missing data point well. If @ is the 
regularization parameter used to obtain X;eg, then it can be shown [36] that the 
GCV method chooses @ to minimize the function 


|b— Axe" 


(trace (1 = AAt.)). 


Where Abe is the matrix such that Xjeg = Alb. For example, in the case of 
Tikhonov regularization (12), 


G(a) = 


At. = (A7A+ 71) 1A’. 
A weighted version of GCV, W-GCV, finds a regularization parameter to 
minimize 


|b — Axreg|” 


(trace (1 — wAAl..)) - 


W-GCV is sometimes more effective at choosing regularization parameters than 
the standard GCV function for certain classes of problems. Setting the weight 
@ = | gives the standard GCV method, while m < 1 produces less smooth 
solutions and w > | produces smoother solutions. Further details about W-GCV 
can be found in [21]. 


Gy (a) = 
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e L-Curve. This approach attempts to balance the size of the discrepancy (i.e., 
residual) produced by the regularized solution with the size of the solution. In 
the context of Tikhonov regularization, this can often be found by a log-log scale 
plot of ||b — AX;eg||2 versus ||Xreg||2 for all possible regularization parameters. 
This plot often produces an L-shaped curve, and the solution corresponding to 
the corner of the L indicates a good balance between discrepancy and size of 
the solution. This observation was first made by Lawson and Hanson [63] and 
later studied extensively, including efficient numerical schemes to find the corner 
of the L (i.e., the point of maximum curvature) by Hansen [46, 52]. Although 
the L-curve tends to work well for many problems, some concerns about its 
effectiveness have been reported in the literature; see [44, 98]. 


There exist many other parameter choice methods besides the ones discussed above; 
for more information, see [28, 49, 99] and the references therein. 

A proper choice of the regularization parameter is critical. If the parameter is 
chosen too small, then too much noise will be introduced in the computed solution. 
On the other hand, if the parameter is too large, the regularized solution may become 
over smoothed and may not contain as much information about the true solution as 
it could. However, it is important to keep in mind that no parameter choice method 
is “fool proof,’ and it may be necessary to solve the problem with a variety of 
parameters and to use knowledge of the application to help decide which solution is 
best. 


Separable Inverse Problems 


Separable nonlinear inverse problems, 
b= A (xeacs) Koc + 9 (20) 


arise in many imaging applications, such as blind deconvolution (see section “Multi- 
frame Blind Deconvolution”), super resolution (which is an example of image 
data fusion) [18, 27,57, 76], the reconstruction of 3D macromolecular structures 
from 2D electron microscopy images of cryogenically frozen samples (Cryo-EM) 
[22, 35, 55, 66, 82, 88], and seismic imaging applications [40]. One could consider 
Eq. (20) as a general nonlinear inverse problem and use the approaches discussed in 
section “Nonlinear Inverse Problems” to compute regularized solutions. However, 
this section considers approaches that exploit the separability of the problem. 
In particular, some of the regularization methods described in section “Linear 
Problems,” such as variational and iterative regularization, can be adapted to 
Eq. (20). To illustrate, consider the general Tikhonov regularized least squares 
problem: 
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A(x) ] @  [b 
Pa P’-[e 


Three approaches to solve this nonlinear least squares problem are outlined in this 
section. 


2 
min {\|A (x) x — pll3 + 02 |[x|]3} =) ity 


x) x(n) x(O) x(n) 


2 
(21) 


Fully Coupled Problem 
The nonlinear least squares problem given in Eq. (21) can be rewritten as 


1 
min g(x) = min = ||p(%)|[3. (22) 


where 


, A (x) b x 
= (© yf) — (€) _ = 
peQH=pixs’= 2") = = x o|: and x= ir) 


Nonlinear least squares problems are solved iteratively, with algorithms having the 
general form: 


Algorithm 1: General Iterative Algorithm 


(6) 
1 Choose initial xp = | 2 | 
Xo 

2 fork =0,1,2,... 

¢ Choose a step direction, dy 

¢ Determine step length, t; 

¢ Update the solution: x,4.) = xx + Tdx 

¢ Stop when a minimum of the objective is obtained 


end 


Typically, d; is chosen to approximate the Newton direction, 


de =—($" Om) 9 Ou), 


where $” is an approximation of $”,/ = a p and Jg is the Jacobian matrix 


i,= E (x, x") ap | 


dxO Ox" 


In the case of the Gauss-Newton method, which is often recommended for 
nonlinear least squares problems, ¢” = Jj Jo- 
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This general Gauss-Newton approach can work well, but constructing and 
solving the linear systems required to update dx can be very expensive. Note that the 
dimension of the matrix Jy corresponds to the number of pixels in the image, x, 
plus the number of parameters in x”, and thus J ¢ may be on the order of 10° x 10°. 
Thus, instead of using Gauss-Newton, it might be preferable to use a low-storage 
scheme such as the (nonlinear) conjugate gradient method. But there is a trade off — 
although the cost per iteration is reduced, the number of iterations needed to attain 
a minimum can increase significantly. 

Relatively little research has been done on understanding and solving the fully 
coupled problem. For example, methods are needed for choosing regularization 
parameters. In addition, the rate of convergence of the linear and nonlinear terms 
may be quite different, and the effect this has on overall convergence rate is not well 
understood. 


Decoupled Problem 

Probably the simplest idea to solve the nonlinear least squares problem is to 
decouple it into two problems, one involving x and the other involving x, 
Specifically, the approach would have the form: 


Algorithm 2: Block Coordinate Descent Iterative Algorithm 


1 Choose initial x 
2 fork =0,1,2,... 
* Choose a; and solve the linear problem: 


2 
® _ F (nt)\ Ve 2 Oy? 
XxX; = argmin | (xf )x-b] +02 |x |, 
x’ 


¢ Solve the nonlinear problem: 


(nf) x 
Xa — ae on 
x 


2 2 
aches — of, + az [xe], 


¢ Stop when objectives are minimized 


end 


The advantage of this approach is that any of the approaches discussed in 
section “Linear Problems,” including methods to determine a, can be used for 
the linear problem. The nonlinear problem involving x“ requires using another 
iterative method, such as the Gauss-Newton method. However, there are often 
significantly fewer parameters than in the fully coupled approach discussed in the 
previous subsection. Thus, a Gauss-Newton method to update x at each iteration 
is significantly more computationally tractable. A disadvantage to this approach, 
which is known in the optimization literature as block coordinate descent, is that 


it is not clear what are the practical convergence properties of the method. As 
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mentioned in the previous subsection, the rate of convergence of the linear and 
nonlinear terms may be quite different. Moreover, if the method does converge, 
it will typically be very slow (linear), especially for problems with tightly coupled 
variables [77]. 


Variable Projection Method 

The variable projection method [38,39,58,79, 86] exploits structure in the nonlinear 
least squares problem (21). The approach exploits the fact that ¢(x, x) is linear 
in x and that x contains relatively fewer parameters than x“. However, rather 
than explicitly separating variables x and x” as in coordinate descent, variable 
projection implicitly eliminates the linear parameters x, obtaining a reduced cost 
functional that depends only on x“, Then, a Gauss-Newton method is used to solve 
the optimization problem associated with the reduced cost functional. Specifically, 
consider 


v (x) =¢ (x© (x) a) 
where x (x) is a solution of 


2 


: () ynt)) _ 
min x’,xX = min 
x 6 ( ) x) 


(23) 


Pe pee H 


To use the Gauss-Newton algorithm to minimize the reduced cost functional 
v(x), it is necessary to compute w/(x""). Note that because x solves (23), 


it follows that we = 0 and thus 
x 


2 


dx 0¢ Ip ao 
dy dx * dx) ~— Ox(nd 


WO) = = Shp, 


where the Jacobian of the reduced cost functional is given by 


0 (A (x9) x) 
Jy = ax) 

Thus, a Gauss-Newton method applied to the reduced cost functional has the basic 
form: 

Although computing Jy is nontrivial, it is often much more tractable than 
constructing Jg. In addition, the problem of variable convergence rates for the two 
sets of parameters, x? and x”), has been eliminated. Another big advantage of 
the variable projection method for large-scale inverse problems is that standard 
approaches, such as those discussed in section “Linear Problems,” can be used to 
solve the linear regularized least squares problem at each iteration, including the 
schemes for estimating regularization parameters. 
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Algorithm 3: Variable Projection Gauss-Newton Algorithm 


1 Choose initial x? mf 


2 fork =0,1,2,... 
3 Choose ax, 
(nl) 
4 x? = arg min . (x! ) xO — H 
x) al 04], 


r=b—-A (7?) x 


d, = arg min ||Jyd — rx||2 


(nf) (n£) 


5 

6 

7 Determine step length rt, 
8 Xp, = X + dk 

9 


end 


Nonlinear Inverse Problems 


Developing regularization approaches for general nonlinear inverse problems can 
be significantly more challenging than the linear and separable nonlinear case. 
Theoretical tools such as the SVD that are used to analyze ill-posedness in the 
linear case are not available here, and previous efforts to extend these tools to 
the nonlinear case do not always apply. For example, a spectral analysis of the 
linearization of a nonlinear problem does not necessarily determine the degree of ill- 
posedness for the nonlinear problem [30]. Furthermore, convergence properties for 
nonlinear optimization require very strict assumptions that are often not realizable 
in real applications [28,29]. Nevertheless, nonlinear inverse problems arise in many 
important applications, motivating research on regularization schemes and general 
computational approaches. This section discusses some of this work. 
One approach for nonlinear problems of the form 


F(x) =b (24) 
is to reformulate the problem to find a zero of F(x) — b = 0. Then a Newton-like 


method, where the nonlinear function is repeatedly linearized around the current 
estimate, can be written as 


Xk4+1 = Xk + PEPE (25) 

where px, solves the Jacobian system 
J(xx)p = b— F(x). (26) 
Though generally not symmetric, matrix and matrix-transpose multiplications with 


the Jacobian, whose elements are the first derivatives of F(x), are typically 
computable. However, the main disadvantages of using this approach are that the 
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existence and uniqueness of a solution are not guaranteed and the sensitivity of 
solutions depends on the conditioning of the Jacobian. Furthermore, there is no 
natural merit function that can be monitored to help select the step length, p;. 
Another approach to solve (24) is to incorporate prior assumptions regarding 
the statistical distribution of the model and maximize the corresponding likelihood 
function. For example, an additive Gaussian noise model assumption under certain 
conditions corresponds to solving the following nonlinear least squares problem: 


min 5 ||b — F(x)|[3. (27) 


Since this is a standard nonlinear optimization problem, any optimization algorithm 
such as a gradient descent or Newton approach can be used here. For problem (27), 
the gradient vector can be written as g(x) = J(x)’ (F(x) — b), and Hessian matrix 
can be written as H(x) = J(x)’ J(x) + Z(x), where Z(x) includes second derivatives 
of F(x). The main advantage of this approach is that a variety of line search methods 
can be used. However, the potential disadvantages of this approach are that the 
derivatives may be too difficult to compute or that negative eigenvalues introduced in 
Z(x) may cause problems in optimization algorithms. Some algorithms for solving 
nonlinear optimization problems are direct extensions of the iterative methods 
described in section “Iterative Regularization.” The nonlinear Landweber iteration 
can be written as 


X41 = Xe + I(x)! (b— F(x), (28) 


which reduces to the standard Landweber iteration if F(x) is linear, and it can 
be easily extended to other gradient descent methods such as the steepest descent 
approach. Newton and Newton-type methods are also viable options for nonlinear 
optimization, resulting in iterates (25) where p; solves 


H(x;,)p = —g (xx). (29) 


Oftentimes, an approximation of the Hessian is used. For example, the Gauss- 
Newton algorithm, which takes H ~ J(x;)’J(x,), is a preferred choice for 
large-scale problems because it ensures positive semi-definiteness, but it is not 
advisable for large residual problems or highly nonlinear problems [40]. Addition- 
ally, nonlinear conjugate gradient, truncated-Newton, or quasi-Newton methods, 
such as LBFGS can be good alternatives if storage is a concern. It is important 
to remark that finding a global minimizer for a nonlinear optimization problem 
is in general very difficult, especially since convexity of the objective function is 
typically not guaranteed, as in the linear case. Thus, it is very likely that a descent 
algorithm may get stuck in one of many local minima solutions. 

When dealing with ill-posed problems, the general approach to incorporate 
regularization is to couple an iterative approach with a stopping criteria such as the 
discrepancy principle to produce reasonable solutions. In addition, for Newton-type 
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methods it is common to incorporate additional regularization for the inner system 
since the Jacobian or Hessian may become ill-conditioned. For example, including 
linear Tikhonov regularization in (26) would result in 


(I(xx)" I(x) + 0°) p = I(x)" (b — F(xx)), 


which is equivalent to a Levenberg-Marquardt iterate, where the update, p;, is the 
solution of a particular Tikhonov minimization problem: 


min || F(x) + J(xe)p — Bll) + @ [pll2. 


where F(x) has been linearized around x;. Other variations for regularizing the 
update can be found in [28] and the references therein. Regularization for the 
inner system can also be achieved by solving the inner system inexactly using 
an iterative method and terminating the iterations early. These are called inexact 
Newton methods, and the early termination of the inner iterations is a good way not 
only to make this approach practical for large-scale problems but also to enforce 
regularization on the inner system. 

The variational approaches discussed in section “Variational Regularization and 
Constraints” can be extended for the second class of algorithms where a likelihood 
function results in a nonlinear optimization problem. For example, after selecting a 
regularization operator 7 (x) and regularization parameter a for (27), the goal would 
be to solve a nonlinear optimization problem of the form 


min { Ib — F(x)||3 + a 7(x)} (30) 


The flexibility in the choice of the regularization operator is nice, but selecting a 
good regularization parameter a priori can be a computationally demanding task, 
especially for large-scale problems. Some work on estimating the regularization 
parameter within a constrained optimization framework has been done [40, 41], 
but the most common approach for regularization of nonlinear ill-posed inverse 
problems is to use standard iterative methods to solve (27), where regulariza- 
tion is obtained via early termination of the iterations. It cannot be stressed 
enough that when using any iterative method to solve a nonlinear inverse problem 
where the regularization is not already incorporated, a good stopping iteration 
for the outer iteration that serves as a regularization parameter is imperative. 
See also [2, 28, 29, 32,54, 92,97] for additional references on nonlinear inverse 
problems. 


4 Numerical Methods and Case Examples 
Given a specific large-scale inverse problem from an imaging application, it can 


be nontrivial to implement the algorithms and regularization methods discussed in 
this chapter. Efficient computations require exploiting the structure of the problem. 
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Moreover, choosing specific regularization schemes and constraints requires knowl- 
edge about the physical process underlying the data collection process. A few 
illustrative examples, using the imaging applications described in section “Imaging 
Applications,” are given in this section. 


Linear Example: Deconvolution 


Perhaps the most well-known and well-studied linear inverse problem is deconvolu- 
tion. As discussed in section “Image Deblurring and Deconvolution,” this spatially 
invariant image deblurring problem is modeled as 


b= AXexact + 7, 


where A is a structured matrix that depends on the PSF and imposed boundary 
conditions. For example, if periodic boundary conditions are imposed on the 
blurring operation, then A has a circulant structure, and moreover, A has the spectral 
decomposition 


A = F*AF, 


where F is a matrix representing a d-dimensional discrete Fourier transform, which 
satisfies F*F = I. The matrix F does not need to be constructed explicitly. Instead, 
fast Fourier transform (FFT) functions can be used to implement matrix-vector 
multiplications with F and F*. Specifically, for 2D images: 


Fx < fft2(x) (2D forward FFT) 
F*x © ifft2(x) (2D inverse FFT) 


The main advantages are that FFT-based spectral filtering regularization algorithms 
are very easy to implement and extremely efficient; see [51] for implementation 
details. 

To illustrate, consider the image data shown in Fig.4, where the simulated 
observed image was obtained by convolving the PSF with the true image and adding 
1% Gaussian white noise. The PSF was constructed from a Gaussian blurring 
operator, 


—G —k)*s3— G —D* sf + 26 -—HU —Dsy 
Pij = exp ( 2 7) : mn s (31) 
287585 — 283 
centered at (k,/) (location of point source), with s; = sz = 5 and s3; = 0. An 
FFT-based Tikhonov spectral filtering solution was computed, with regularization 
operator L = I and regularization parameter a = 0.00544, which was chosen 


using GCV. (All computations for this example were done with MATLAB. The 
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Fig. 4 Simulated data for an 

image deconvolution 

problem. The restored image 

was computed using an 

FFT-based spectral filtering 

method, with Tikhonov ».- 

regularization and 

GCV-chosen regularization 

parameter 


Truth PSF 
Ea Xx 
Blurred Restored 


implementation of the FFT-based spectral filter used in this example is described 
in [51]. The MATLAB code, which is called tik_fft1m, can be found at http:// 
www2.imm.dtu.dk/~pch/HNO/#.) The reconstructed image, which was computed 
in a fraction of a second on a standard laptop computer, is also shown in Fig. 4. 

If there are significant details near the boundary of the image, then the peri- 
odic boundary condition assumption might not be an accurate representation 
of the details outside the viewable region. In this case, severe ringing artifacts 
can appear in the reconstructed image, and parameter choice methods may per- 
form very poorly in these situations. Consider, for example, the image data 
shown in Fig.5. The PSF is the same as in the previous example, but the 
blurred image contains features at the boundaries of the viewable region. The 
“restored” image in Fig.5 was again computed using a Tikhonov spectral fil- 
tering solution with regularization operator L = I, and regularization param- 
eter (@ = 6.30 x 107°) was chosen using GCV. This noise-corrupted recon- 
structed image indicates that the regularization parameter chosen by GCV is too 
small. 

It is possible that another parameter choice method would perform better, but 
it is also the case that imposing alternative boundary conditions may improve the 
situation. For example, reflective boundary conditions assume the image scene 
outside the viewable region is a mirror image of the details inside the viewable 
region. With this assumption, and if the PSF is also circularly symmetric, then the 
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Fig. 5 Simulated data for an 
image deconvolution 
problem. The restored image 
was computed using an 
FFT-based spectral filtering 
method, with Tikhonov 
regularization and 
GCV-chosen regularization 
parameter 


Truth PSF 


Blurred Restored 


matrix A has a symmetric Toeplitz-plus-Hankel structure, and, moreover, A has the 
spectral decomposition 


A=C'AC, 


where C is a matrix representing a d-dimensional discrete cosine transform, which 
satisfies C’C = I. As with FFTs, the matrix C does not need to be constructed 
explicitly, and very efficient functions can be used to implement matrix-vector 
multiplications with C and C’, such as 


Cx > dct2(x) (2D forward DCT) 
C’x < idct2(x) (2D inverse DCT) 


In addition, DCT-based spectral filtering regularization algorithms are very easy to 
implement and are extremely efficient; see [51] for implementation details. 

Figure 6 illustrates the superior performance that can be obtained if the boundary 
condition and the corresponding basis (in this case, DCT) is used in the spectral 
filtering deconvolution algorithms. Specifically, the image on the left in Fig. 6 was 
computed using a DCT-based Tikhonov spectral filtering method, with regulariza- 
tion operator L = I and a GCV-chosen regularization parameter a = 4.83 x 107°. 
The image on the right was computed using the FFT-based Tikhonov filter, but 
instead of using the GCV-chosen regularization parameter (which produced the 
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DCT, a= 4.83 x 107% FFT, @=6.30 x 10°5 FFT, «@=4.83 x 103 


Fig. 6 These restored images were computed using DCT and FFT-based spectral filtering 
methods, with Tikhonov regularization. For the DCT and the middle FFT reconstructions, the 
regularization parameter @ was chosen by GCV. The FFT reconstruction on the right was obtained 
using the same regularization parameter as was used for the DCT reconstruction 


Truth PSF 


Blurred 


Fig. 7 Simulated deconvolution data with a nonsymmetric Gaussian PSF 


poor reconstruction displayed in the middle of this figure), @ was set to the same 
value used by the DCT reconstruction. This example clearly illustrates that the 
quality of the reconstruction, and the effectiveness of parameter choice methods, 
can depend greatly on the imposed boundary conditions and corresponding spectral 
basis. (As with previous examples, all computations described here were done with 
MATLAB. The implementation of the DCT-based spectral filter is described in [51]. 
The MATLAB code, which is called tik_dct.m, can be found at 
) 

Spectral filtering methods work well for many deconvolution problems, but it 
may not always be possible to find a convenient basis that allows for efficient 
implementation. Consider, for example, the data shown in Fig. 7. The PSF in this 
figure was constructed using Eq. (31), with s} = s2=3 and 53 =2, and results in a 
nonsymmetric PSF. As with the previous example, the FFT-based filter does not 
work well for this deconvolution problem because of its implicit assumption of 
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Fig. 8 These restored images 
were computed using a 
DCT-based spectral filtering 
method (/eft) and an iterative 
hybrid method (right) 


DCT, @=4.83 x 10°3 HyBR, a= 7.31 x 10° 


periodic boundary conditions. Reflective boundary conditions are more appropriate, 
but the lack of circular symmetry in the PSF means that the DCT basis does not 
diagonalize the matrix A. The reconstructed image on the left in Fig. 8 illustrates 
what happens if we attempt to reconstruct the image using a DCT-based Tikhonov 
filter. 

An iterative method may be the best option for a problem such as this; one can 
impose any appropriate boundary condition (which only needs to be implemented 
in matrix-vector multiplications with A and A’) without needing to assume any 
symmetry or further structure in the PSF. The reconstructed image shown on the 
right in Fig. 8 was obtained using a hybrid approach described in section “Hybrid 
Iterative-Direct Regularization.” Specifically, Tikhonov regularization is used for 
the projected subproblem, with regularization parameters chosen by W-GCV. The 
MATLAB software for this, which is called HyBR, is discussed in [21] and 
can be obtained from http://www.math.vt.edu/people/jmchung/hybr.html. For this 
particular example, HyBR terminated at iteration 21, with a regularization parameter 
a = 7.31 x 107°. 

The examples in this subsection illustrate that many approaches can be used 
for the linear inverse problem deconvolution. It is possible that other methods, 
such as those that incorporate nonnegativity constraints, may produce better results 
than those presented here, but this is typical of all inverse problems. It would be 
impossible to give an exhaustive study and comparison in this chapter. 


Separable Example: Multi-Frame Blind Deconvolution 


In this section, multi-frame blind deconvolution (MFBD) is used to illustrate a 
numerical example of a separable (nonlinear) inverse problem, 


£ L 
b=A (x2) a + 9. 


Recall from section “Multi-frame Blind Deconvolution” that in MFBD, a set of, 
say, m blurred images of an object are collected, and the aim is to simultaneously 
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~~ WR 


Fig. 9 Simulated MFBD data. The images were obtained by convolving the true satellite image 
from Fig. 4 with Gaussian PSFs using parameters given in Eq. (32) and then adding 1 % white 
noise 


reconstruct an approximation of the true image as well as the PSFs (or the 
parameters that define the PSFs) associated with each of the observed blurred 
images. Such an example can be simulated using the Gaussian blurring kernel given 
in Eq. (31), and the true satellite image given in Fig. 4. Specifically, suppose using 
Eq. (31), three PSFs are constructed using the following values: 


6.0516 
5.8419 
2.2319 
5.4016 
x") = | 4.3802 
2.1562 
5.7347 
6.8369 
2.7385 


Gaussian PSF parameters $1, 52, 53 for frame | 


Gaussian PSF parameters 51, 52, 53 for frame 2 (32) 


Gaussian PSF parameters $1, 52, 53 for frame 3 


Simulated observed image data can then be generated by convolving the PSFs 
constructed from these sets of parameters with the true satellite image and then 
adding 1 % white noise. The resulting simulated observed image frames are shown 
in Fig. 9. 

Image reconstructions can then be computed using the variable projection Gauss- 
Newton algorithm described in section “Variable Projection Method.” The Jacobian 
Jy can be constructed analytically for this problem (see, e.g., [19]), but a finite 
difference approach can also work very well. In the experiments reported here, 
centered differences were used to approximate the Jacobian. 

The hybrid method implementation HyBR, described in the previous subsection, 
was used to choose a; and to solve the linear subproblem for x”. The step length 


(ne) 
0 


T, was chosen using an Armijo rule [77]. The initial guess for x,“ was 
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Fig. 10 Convergence results for MFBD. The relative error of the estimated PSF parameters at 


each iteration is shown in the left plot, while the relative error of the reconstructed image at each 
iteration is shown in the right plot 


Fig. 11 On the /eft is the initial reconstructed image using x) and on the right is the final 
reconstructed image. The true image is displayed in the middle for comparison purposes 


7.0516 a 
7.8369 initial guess for 51, 52, 53 for frame | 
3.2385 
7.0516 

xi" = | 7.8369 initial guess for 51, 52, 53 for frame 2 
3.2385 
7.0516 
7.8369 initial guess for 51, 52, 53 for frame 3 
3.2385 


The results in Fig. 10 show the convergence behavior in terms of relative error at 
each iteration of the variable projection Gauss-Newton algorithm for this example. 
The left plot shows the convergence history of x and the right plot shows the 
convergence history of xO, Note that the convergence behavior of both terms is very 
similar. Figure 11 shows the reconstructed image after the first variable projection 
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Gauss-Newton iteration (i.e., the initial reconstruction) and the reconstructed image 
after the last iteration of the algorithm. 


Nonlinear Example: Tomosynthesis 


Polyenergetic digital tomosynthesis is an example of a nonlinear inverse problem 
where the forward problem can be modeled as (8). The typical approach to compute 
an approximation of Xexact is to assume that the observed projection image is a 
realization of a Poisson random variable with mean values 


Ne 


by + 7 = )_ ofc) exp(—[s(e)Aox + z(e)Agl]) + i, (33) 


e=1 


where 7 is the mean of the additive noise. Then the maximum likelihood estimator 
(MLE) can be found by minimizing the negative log likelihood function: 


M 
— Lo(x) = > (5) + i) - bi? log (5) + i) +c, (34) 
i=l 
where superscripts refer to entries in a vector and c is a constant term. A regularized 
estimate can be found by solving the following nonlinear optimization problem 


n@ 


XMLE = min | > —Lo(x) 


6=1 


(35) 


using a gradient descent or Newton-type algorithm and terminating the iterations 
before the noise enters the problem. For this example, the gradient of the objective 
function with respect to the 3D volume, x, can be written as 


g(xe) = Al vy 


where the entries of vector v, are given by 


© pr -1) Soe)o(e) exp-Is(e)al ne + (eal) 
= | —__ — exp(— XE ‘1)). 
v 504 RO ee p(—[s(e)a; xx + z(e)a; 
The Hessian matrix can be written as 


H, = A’ W,A 


where W, is a diagonal matrix with vector w, on the diagonal. A mathematical 
formula for the values of the diagonal can be quite complicated as they depend on 
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Fig. 12 Extracted regions of sample projection images 


the values of the second derivatives. Furthermore, the Newton step at iteration k in 
Eq. (29) is just the normal equations formulation of the least squares problem 


1 


f a 
W? Asx — W, * vi (36) 


min 
Sk 


2 


1 1 
where WwW? = diag (w). For solving the Newton system, CGLS can be used to 


solve (36) inexactly. Furthermore, regularization for the outer problem is achieved 
by early termination of the iterative optimization method. 

The example illustrated here comes from a true volume of size 128 x 128 x 128 
whose values range between 0 and 100, representing the percentage of glandular 
tissue present in the voxel. Then 21 projection images were taken at equally spaced 
angles, within an angular range from —30° to 30° at 3° intervals, using the typical 
geometry for breast tomosynthesis, illustrated in Fig. 1. Each 2D projection image 
contains 150 x 200 pixels. Subimages of three of these projections can be found in 
Fig. 12. 

The original object represented a portion of a patient breast with mean com- 
pressed breast thickness of size 6.4 x 6.4 x 6.4.cm, and the detector was 7.5 x 10cm. 
The source to detector distance at 0° was set to 66cm, and the distance from the 
center of rotation to detector was 0 cm. The incident X-ray spectrum was produced 
by a rhodium target with a tube voltage of 28 kVp and an added rhodium filter of 
25 1m thickness, discretized to consist of 47 different energy levels, from 5.0 to 
28 keV, in 0.5 keV steps. 

For the reconstruction algorithms, the ray trace matrix Ag for each projection 
angle was computed using a cone beam model, and an initial guess of the volume 
was a uniform image with all voxel values set to 50, meaning half glandular and 
half adipose tissue. The reconstructed volume consisted of 128 x 128 x 40 voxels 
with a voxel size of 500 x 500 tm x 1.6mm. Furthermore, additive Poisson noise 
was included in the projection images so that there was a relative noise level of 
approximately | %. Some slices of the true volume can be found in Fig. 13. 
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Fig. 13 Sample slices from the original breast volume 


Table 1 Convergence results for polyenergetic tomosynthesis reconstruction 


Gradient descent method 


Iteration Rel. objective Rel. gradient Rel. error 

0 7.033 — 04 1.0000 0.4034 

1 6.771e — 04 0.8755 0.3562 

3 6.586e — 04 0.2731 0.2948 

10 6.585e — 04 0.0641 0.2762 

25 6.55 le — 04 0.0314 0.2386 

50 6.548e — 04 0.0104 0.2237 

Newton-CG 

Iteration Rel. objective Rel. gradient Rel. error CGLS iterations 
0 7.033 — 04 1.0000 0.4034 - 
1 6.587e — 04 0.2525 0.2814 5 
2 6.550e — 04 0.0398 0.2293 9 
3 6.547e — 04 0.0065 0.2075 22 
4 6.547e — 04 0.0013 0.2014 50 
P) 6.547e — 04 0.0009 0.2003 50 


Recall that the goal of digital tomosynthesis is to reconstruct an approximation 
of the 3D volume, x, given the set of projection images bg, 9 = 1,2,...,¢. Using 
the above likelihood function, the problem has been reformulated as a nonlinear 
optimization problem for which standard numerical optimization schemes can be 
applied. A gradient descent, Newton-CG, and LBFGS algorithm are investigated 
as methods to solve this problem, and early termination of the iterative method 
produces a regularized solution. 

Results presented in Table | include the iteration, the relative objective function 
value, the relative gradient value, and the relative error for the 3D volume for two 
iterative algorithms. The relative error can be computed as Ba where x, is 
the reconstructed volume at the kth iteration. For the inexact Newton-CG algorithm, 
the stopping criterion used for CGLS on the inner problem (36) was a residual 
tolerance of 0.17 and a maximum number of 50 iterations. The number of CGLS 
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iterations reported for the inner problem at each Newton-CG iteration can be found 
in the last column of the table. It is worth mentioning here that many parameters 
such as the number of inner and outer iterations rely heavily on heuristics that may or 
may not be provided by the application. In any case, appropriate parameters should 
be used in order to ensure that nonlinearity and ill-posedness of the problem are 
addressed. 

Since each Newton-CG iteration requires the solution of a linear system, it is 
difficult to present a fair comparison of reconstruction algorithms. In terms of 
computational effort, the most computationally burdensome aspect of the recon- 
struction is the matrix-vector and matrix-transpose-vector multiplications with ray 
trace matrix, A. Each function and gradient evaluation of the likelihood function 
requires a total of three “ray trace” multiplications (two for the function evaluation 
and one more for the gradient), and a multiplication operation with the Hessian 
(or its transpose) only requires two “ray trace” multiplications. Furthermore, a 
backtracking line search strategy is used to ensure sufficient descent at each iteration 
of the optimization scheme. The Cauchy point [77] is used as an initial guess in 
the line search scheme, thus requiring another multiplication with the Hessian. 
Thus, the computational cost and timing for, say, one Newton-CG iteration with 
50 inner CG iterations with the Hessian is not equivalent to 50 gradient descent 
iterations. 

Another important remark is that although the image errors in Table | decrease 
in the early iterations, these errors eventually increase. This is illustrated in the 
later Newton-CG iterations in Fig. 14, where plots of the relative errors per iteration 
for the three algorithms are presented. From Fig. 14, it is evident that the gradient 
descent method is slow to converge. On the contrary, Newton methods can compute 
a good approximation very quickly, but corruption from errors occurs quickly as 
well. Although LBFGS is typically used for problems where the Hessian cannot 
be computed directly, this approach seems to offer a good balance between fast 
convergence and slow semi-convergence behavior. An important remark is that 
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True Gradient Newton-CG 


Fig. 15 Comparison of slices from the reconstructed volumes computed after 3 iterations of 
Newton-CG algorithm and 15 iterations of gradient descent 


direct regularization techniques can also be used to regularize the problem, but 
appropriate regularization operators and good regularization parameter selection 
methods for this problem are still topics of current research. Thus, regularization 
via early termination of the iterations is the approach followed here. 

For a comparison of images, Fig. 15 contains corresponding slices from the 
reconstructed volumes after three Newton-CG iterations and 15 gradient descent 
iterations, each requiring approximately 80 matrix-vector operations. It is evident 
that the Newton-CG reconstruction has more fine details and more closely resembles 
the true image slice. 

Although nonlinear inverse problems can be difficult to analyze, there are a 
variety of scientific applications such as polyenergetic digital breast tomosynthesis 
that require methods for computing approximate solutions. Iterative methods with 
regularization via early termination can be a good choice, but proper precondition- 
ing techniques may be needed to accelerate the algorithms and good heuristics are 
required. 


5 Conclusion 


Large-scale inverse problems arise in many imaging applications. The examples in 
this chapter illustrate the range of difficulties (from linear to nonlinear) that can 
be encountered and the issues that must be addressed when designing algorithms. 
It is important to emphasize that the literature in this field is vast and that this 
presentation is far from being a complete survey. However, the techniques discussed 
in this chapter can be used as a foundation on which to learn more about the subject. 

The study of inverse problems continues to be an extremely active field of 
research. Although linear inverse problems have been fairly well studied, some 
fundamental questions still need to be addressed and many open problems remain. 
For example, in hybrid algorithms, simple filtering methods (e.g., truncated SVD 
or standard Tikhonov regularization) and standard regularization parameter choice 
methods (e.g., discrepancy principle or GCV) are typically used to regularize the 
projected problem. Some work has been done to generalize this (see, e.g., [59]), 
but extensions to more sophisticated filtering algorithms and parameter choice 
methods should be investigated. In addition, the development of novel algorithmic 
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implementations and software is necessary for running existing algorithms on 
state-of-the-art computing technologies, as is the development of techniques for 
uncertainty quantification. Another area of active research for the solution of 
linear and nonlinear inverse problems is sparse reconstruction schemes, where 
regularization enforces some structure to be sparse in a certain basis, that is, 
represented with only a few nonzero coefficients. 

As discussed in sections “Separable Inverse Problems” and “Nonlinear Inverse 
Problems,” there are many open problems related to solving nonlinear inverse 
problems. For example, in the case of the variable projection Gauss-Newton method, 
a thorough study of its regularization and convergence properties remains to be 
done, especially in the context of an iteration-dependent regularization parameter. 
For more general nonlinear problems, issues that need to be addressed include ana- 
lyzing the sensitivity of the Jacobian and Hessian matrices, as well as determining 
appropriate merit functions for selecting step lengths. In nonlinear optimization, 
difficulties arise because convexity of the objective function cannot be guaranteed, 
so algorithms can become trapped in local minima. More work also needs to be 
done in the area of regularization parameter choice methods for nonlinear problems 
and appropriate stopping criteria for iterative methods. For a further discussion of 
open problems for nonlinear inverse problems, see [28,29]. 

Finally, it should be noted that many open problems are given in the context 
of the application, such as determining appropriate constraints and regularization 
operators for the problem. Future directions are often motivated by the application, 
and many of these questions can be found in application-specific references; see, 
for example, [17]. With such varied and widespread applications, large-scale inverse 
problems continue to be a thriving research interest in the mathematics, computer 
science, and image processing communities. 
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Abstract 


In this chapter are outlined some aspects of the mathematical theory for direct 
regularization methods aimed at the stable approximate solution of nonlinear ill- 
posed inverse problems. The focus is on Tikhonov type variational regularization 
applied to nonlinear ill-posed operator equations formulated in Hilbert and 
Banach spaces. The chapter begins with the consideration of the classical 
approach in the Hilbert space setting with quadratic misfit and penalty terms, 
followed by extensions of the theory to Banach spaces and present assertions 
on convergence and rates concerning the variational regularization with general 
convex penalty terms. Recent results refer to the interplay between solution 
smoothness and nonlinearity conditions expressed by variational inequalities. 
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Six examples of parameter identification problems in integral and differential 
equations are given in order to show how to apply the theory of this chapter to 
specific inverse and ill-posed problems. 


1 Introduction 


This chapter will be devoted to direct regularization methods — theory and examples 
— for the solution of inverse problems formulated as nonlinear ill-posed operator 
equations 


F(x)=y, (1) 


where the forward operator F : D(F) C X — Y with domain D(F’) maps between 
infinite dimensional normed linear spaces X and Y, which are Banach spaces or 
Hilbert spaces, with norms || - ||. The symbol (-, -) designates inner products in 
Hilbert spaces and dual pairings in Banach spaces. Moreover, the symbols — and — 
denote the norm convergence and weak convergence, respectively, in such spaces. 
It is well known that the majority of inverse problems are ill-posed in the sense of 
Hadamard, i.e., at least one of the following difficulties occurs: 


(i) Equation (1) has no solution in D(F) if the exact right-hand side y is replaced 
by a perturbed element y® (noisy data) satisfying the inequality 


|v°-yl] <6 (2) 


with noise level 6 > 0. 
(11) The solution to Eq. (1) is not uniquely determined in D(F). 

(iii) The solution to Eq.(1) is unstable with respect to perturbations, i.e., for 
x* € D(F) with F (x°) = y°® and (2) the norm deviation |x? —-Xx | may be 
arbitrarily large. In other words, the possibly multivalued inverse operator F~! 
fails to be continuous. 


Since for nonlinear equations the local behavior of solutions is of main interest, 
the aspect of local ill-posedness according to [64] is focused on numerous consid- 
erations. An operator equation (1) is called locally ill-posed at some solution point 
x! € D(F) if for any ball B, (x*) with center x‘ and an arbitrarily small radius 
p > 0 there exist infinite sequences {x,} C B, (x) N D(F) such that 


F(x,) > F (x") in Y, but a inX asn—>oo. 


In case of local ill-posedness, x‘ cannot be identified arbitrarily precise by noisy 
data y® even if the noise level 4 is arbitrarily small. The aspect of local ill-posedness 
involves both the non-injectivity of F around x‘ corresponding with (ii) and the 
local instability of (1) corresponding with (iii) in Hadamard’s sense. Wide classes 
of inverse problems that have smoothing, for example, compact, forward operators 
F lead to locally ill-posed situations. 
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To overcome the ill-posedness and local ill-posedness of Eq. (1), in particular 
to compensate the instability of solutions with respect to small changes in the 
right-hand side expressing a deficit of information in the data with respect to the 
solution to be determined, regularization methods have to be used for the stable 
approximate solution of Eq.(1) whenever only noisy data are given. The basic 
idea of regularization is to replace the ill-posed original problem by a well-posed 
and stable neighboring problem. A regularization parameter ~ > O controls the 
trade-off between closeness of the neighboring problem expressed by small values 
a and high stability of the auxiliary problem expressed by large values a. In the 
former case the approximate solutions are too unstable, whereas in the latter case, 
approximate solutions are too far from the original one. On the other hand, the loss 
of information in the data caused by smoothing properties of the forward operator 
F can be diminished when external a priori information is exploited. This can be 
done by the choice of appropriate structure in the neighboring problems. 

If the forward operator F and hence the operator equation (1) is linear, then in 
Hilbert spaces a comprehensive and rather complete regularization theory, including 
a general regularization schema and a well-established collection of methods, 
assertions on stability, convergence, and convergence rates, is available since more 
than 20 years; see [10, 35, 77—79, 84]. For recent progress of regularization theory 
applied to linear ill-posed problems, please refer to the papers [16, 22, 31,57, 61, 
80, 85, 86, 90]. It is well known that inverse problems aimed at the identification 
of parameter functions in differential equations or boundary conditions from 
observations of state variables are in general nonlinear even if the differential 
equations are linear. The nonlinearity of F, however, makes the construction and 
the use of regularization methods more complicated and diversified; please refer 
to [9, 26, 27, 33, 38, 45, 49, 56, 66, 71, 81, 87] for more details. Furthermore, there 
have been recent significant progress in regularization theory for ill-posed problems 
formulated in Banach spaces (see, e.g., [22, 60, 96, 97, 102, 103, 107, 109]). In this 
chapter, the focus is on direct regularization methods for the stable approximate 
solution of nonlinear ill-posed operator equations formulated in Hilbert and Banach 
spaces, where regularized solutions mostly are solutions of variational problems. 
The functional to be minimized over a set of admissible solutions contains a 
regularization parameter a > O which has to be controlled in an appropriate 
manner. An alternative way of regularization is the solution of (1) for noisy data 
y® by an iteration process, where the stopping criterion, frequently depending on 
6, plays the role of the regularization parameter. For iterative solution methods, 
please refer to the corresponding chapter of this book and to the monograph 
[8, 76]. 


2 Theory of Direct Regularization Methods 


In contrast to the classical treatment of linear ill-posed problems, where regularized 


solutions x = R,y’, i.e., stable approximate solutions to Eq. (1) under the noise 


model (2), are obtained by applying bounded linear operators Ry, : Y — X to the 
data y® for regularization parameters a > 0, such explicit approach fails if either 
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in (1) (a) the forward operator F is nonlinear, (b) the domain D(F) is not a linear 
subspace of X, or (c) the mapping y> +> a is continuous but nonlinear for all a > 0 
even if F is linear. All three sources of nonlinearity make it necessary to define 
the regularized solutions in an implicit manner. The preferred approach of direct 
regularization methods is variational regularization, where regularized solutions a 
are minimizers of the functional 


®(x) := S (F(x), y’) + aR(x) (3) 


by assuming that S is a nonnegative misfit functional measuring the discrepancy 
between F(x) and the data y5, moreover w > 0 is the regularization parameter, 
and R with domain D(R) := {x € X : R(x) < co} is a nonnegative 
stabilizing functional with small values for elements x being reliable and large 
values for improbable x. Since the origins of this method go back to the work 
of A. N. Tikhonov and his collaborators (see [114, 115]), this method is often 
called Tikhonov type regularization. For example, please refer to the monographs 
[7, 8, 10, 35, 54, 59, 68, 73, 92, 107, 109, 116, 117] and to the papers [82, 83, 98, 118], 
which contribute to the theory and practice of this kind of regularization. 
Besides the standard version 


S(F(x), 9) =|F@)-y' |’, p=, (4) 


which is mostly used in combination with the noise model (2) in Banach spaces, 
specific noise models like Poisson noise or a stochastic background suggest alterna- 
tive choices for the misfit term S like Kullback-Leibler or other divergences. With 
respect to imaging, e.g., deblurring, image reconstruction, image registration, and 
partial differential equations occurring there, different chapters of the monographs 
(2, 15,39,41,91,94, 106, 107] and the papers [12,23,40,93, 120] motivate and discuss 
regularized solutions x? as well as different choices of functionals S and R. On the 
other hand, the minimizers of (3) play also an important role in the treatment of 
statistical inverse problems by Bayesian methods, maximum a posteriori estimation 
(MAP), and penalized maximum likelihood estimation (see, e.g., [74]), where in 
some cases the penalty term (x) can even be determined by a priori information 
when the solution x is a realization of a randomized state variable. 

A typical property of ill-posed equations is that minimizing S (F (x), y°) alone 
is very sensitive to data changes and yields in most cases highly oscillating 
minimizers. For example, the least-squares approach | F(x) — y® | > _. min as pre- 
ferred method in Hilbert spaces shows such behavior. Therefore, the regularization 
parameter a > 0 in the variational problem ®(x) — min controls the trade-off 
between optimal data fitting with unstable solutions if a is near zero and a high 
level of stability and sympathy but larger misfit for the approximate solution if @ is 
more far from zero. The set of admissible solutions in the process of minimizing ® 
from (3) is the intersection 


D := D(F)ND(R) 
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of the domains of F and 7. For obtaining a regularized solution x! to a nonlinear 
inverse problem, a nonlinear and frequently non-convex optimization problem has 
to be solved, since either the functional ® or the set D(F’) can be non-convex. 
As a consequence, for the numerical treatment of direct regularization methods in 
combination with discretization approaches, iterative procedures are also required. 
In this context, the details have been omitted, and the reader is referred to the 
monographs [25, 35, 107, 116] and to the sample [16, 31,69, 72, 101] of papers from 
a comprehensive set of publications on numerical approaches. 

The appropriate choice of a is one of the most serious tasks in regularization, 
where a priori choices a = a(6) and a posteriori choices a = a (5, y°) have to 
be distinguished. For a priori choices the decay rate of a(S) > 0 as 6 > O is 
prescribed with the goal that regularized solutions converge to a solution of (1), 
i.e., xe) — x' as 8 — 0. Such convergence can be arbitrarily slow depending on 


smoothness properties of x*. To obtain convergence rates [x3 (6) — xt | = O(¢(64)) 


as 6 — 0, that means a uniform convergence for some nonnegative increasing 
rate function g(5) with g(0) = 0, additional conditions on x‘, so-called source 
conditions, have to be satisfied. In contrast to a priori choices, an a posteriori 
choice of the regularization parameter a takes into account the present data y® 


and tries to equilibrate the noise level 6 and the deviation between F (z 6 sis) 


and y®. By the discrepancy principle, as the most prominent approach, @ is 
chosen originally such that | F (Xa(s,y8 )° a xt | = C6 with some constant C > 


1 whenever (4) and (2) are supposed. Various discussions, generalizations, and 
improvements of the discrepancy principle can be found in [3, 5, 69, 92, 113]. If 6 
is not known sufficiently well, then heuristic methods for choosing a = a ( ” ) can 
be exploited as the quasioptimality principle, the L-curve method, and others (see, 
e.g., [7,35,55]). They have theoretical drawbacks, since convergence fails in worst 
case situations, but the utility of those methods for many classes of applications is 
beyond controversy. 

The choices of S, Rk, anda = a (5, y’) should be realized such that the 
following questions Q1—Q4 can be answered in a positive manner: 


Q1: Do minimizers x of the functional ® from (3) exist for all wa > 0 and y e€Y? 

Q2: Do the minimizers x for fixed w > 0 stably depend on the data y°? 

Q3: Is there a convergence x69) — x7 toa solution x" of (1) if under (2) 8 > 0? 

Q4: Are there sufficient conditions imposed on x? for obtaining convergence rates 
||x3 _ xt| = O(9(65)) as 6 > 0? In this context, g : (0,00) > (0, oo) is an 
index function, which means that ¢g is continuous and strictly increasing with 


li = 0. 
papas 


Sometimes the requirement of norm convergence is too strong. If, for instance, the 
penalty functional attains the form R(x) = ||x||4, g > 0, then F is stabilizing 
only in the sense of a topology, which is weaker than the norm topology in X. 
Precisely, the level sets {x € X : R(x) < c} are weakly sequentially compact in X 
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if X is a Hilbert space or a reflexive Banach space. In such case, weak convergence 
a a x? of regularized solutions is a reasonable requirement. This leads to 
norm convergence of regularized solutions if the Radon-Riesz property is satisfied; 
see [107, 109]. On the other hand, it may be useful to replace the norm as a measure 
for the error of regularization by alternative measures EF (xd, xt), preferably the 
Bregman distance if 7e is a convex functional and X is a Banach space; see [22, 41, 


50,58, 60]. 


Tikhonov Regularization in Hilbert Spaces with Quadratic Misfit and 
Penalty Terms 


In Hilbert spaces X and Y, quadratic Tikhonov regularization with the functional 
2 «2 
B(x) = FOX) —y* | + alle — 2" (5) 


to be minimized over D = D(F) is the most prominent variant of variational 
regularization of nonlinear ill-posed operator equations, for which the complete 
theory with respect to questions Q1—Q4 was elaborated 20 years ago (see [37, 110]). 
For a comprehensive presentation including the convergence rates results, please 
refer to [35, Chapter 10]. 

For some initial guess or reference element, x* € X minimizers of (5) tend to 
approximate x*-minimum norm solutions x? to (1) for which 


|x? — x* || = min {|x —x*||: FQ) =y, x € D(F)}. 


Note that x*-minimum norm solutions need not exist. In case of existence they 
need not be uniquely determined. However, under the following assumption, the 
existence of a solution x! € D(F) to (1) implies the existence of an x*-minimum 
norm solution (see [107, Lemma 3.2]). 


Assumption 1. /. The operator F : D(F) © X — Y maps between Hilbert 
spaces X and Y with a nonempty domain D(F). 

2. F is weakly sequentially closed, i.e., weak convergence of the sequences X, — Xo 
and F(x) — yo with x, € D(F), xo € X, yo € Y implies x9 € D(F) and 
F(xX0) = yo- 


For checking item 2 of Assumption 1, it is important to know that in the case 
of weakly closed and convex domains D(F), the weak continuity of F, i.e., 
Xn — Xo implies F(x,) — F (xo), is a sufficient condition. Moreover, the following 
proposition (see [35, Section 10.2]) answers the questions Q1-—Q3 in a positive 
manner. 


Proposition 1. Under Assumption 1 the functional (5) has a minimizer x8 € D(F) 
for alla > Oand y° € Y. For fixed « > 0 and a sequence yn, — y®, every infinite 
sequence {Xx,} of minimizers to the associated functionals 
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®, (x) = [| F(x) — yall? +o [lx — x* ||? (6) 


has a convergent subsequence, and all limits of such subsequences are minimizers 
x® of (5). Whenever the a priori parameter choice a = a(S) > 0 for 5 > 0 satisfies 


2 
a(é) > 0 and —~-—>0, as 6-0, 


a(d) 


and if (1) has a solution in D(F), 6, — 0 is a sequence of noise levels with 


corresponding data y, = y°" such that \|yn — y|| < 8n, then every associated 


sequence {xX,} of minimizers to (6) has a convergent subsequence, and all limit 
elements are x*-minimum norm solutions x* of (1). 


An answer to question Q4 concerning convergence rates is given by the following 
theorem along the lines of [35, Theorem 10.4]. 


Theorem 2. In addition to Assumption 1, let D(F) be convex and x‘ be an x*- 
minimum norm solution to (1) such that 


|| FQ) -— F (x*) —A(x—x")| < 5 Jx—x?]? forall x € D(F) 2 By (x") 


(7) 
holds for a ball B, (x) with sufficiently large radius p > 0, a constant L > 0, and 
a bounded linear operator A: X — Y satisfying a source condition 


xt—x* = A*w, (8) 


where A* : Y — X is the adjoint operator to A and w € Y is some source element 
fulfilling the smallness condition 


L|lwl| <1. (9) 
Then the error estimate 
6 + a||w| 
Val — L|\w| 


and, for the a priori parameter choice cé < a(6) < €6 with constants0 <c <C < 
oo, the convergence rate 


Ixo-x"| s 


xh —x'| = 0 (v8) as 6 >0 (10) 
are obtained. 


The operator A in (7) must be considered as a linearization of F at the point x* in 
the sense of a Gateaux or Fréchet derivative F’ (x‘). The condition (7) characterizes 
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the structure of nonlinearity of the forward operator F in a neighborhood of x". If 
the Fréchet derivative F’(x) exists and is locally Lipschitz continuous around xt 
with Lipschitz constant L > 0, then (7) is fulfilled. 

For further convergence rate results of Tikhonov regularization in Hilbert spaces 
with quadratic penalty term, please refer, for example, to [64,75,88,89,95, 105, 111- 
113]. 


Variational Regularization in Banach Spaces with Convex 
Penalty Term 


In Banach spaces X and Y, the wide variety of variational regularization realized by 
minimizing the functional ® from (3) allows establishing a priori information about 
the noise model and the solution x‘ to be determined in a more sophisticated manner 
than Tikhonov regularization in Hilbert spaces with quadratic misfit and penalty 
terms. Knowledge of the specific situation motivates the selection of the functionals 
S and R, where norm powers (4) are considered here as misfit functional, which, 
for example, simplifies the numerical treatment of minimization problems if Y 
is a Lebesgue space Y = L?(Q) or a Sobolev space Y = W/?(Q) with 
l<p<w, QC IR“. Please refer to the papers [6, 32, 44, 51,53, 63,99] for a 
further discussion of alternative misfit functionals S. In most cases convex penalty 
functionals F are preferred. An important class of penalty functionals form the norm 
powers R(x) := lll. q > 0, x € X, where as an alternative to the standard case 


X = X the space X can also be chosen as a dense subspace of X with stronger 
norm, e.g., X = L1(Q), X = W'4(Q), 1 = 1,2,.... To reconstruct non- 
smooth solutions x‘, the exponent g can be chosen smaller than two, for example, 
close to one or g = | if the solution is assumed to be sparse. To recover solutions 
for which the expected smoothness is low, also penalty terms R(x) = TV(x) are 
frequently applied, partly in a modified manner, where TV(x) = [ |Vx| expresses 


the total variation of the function x (see, e.g., [1, 13, 107, 118, L19)). For specific 
applications in imaging (see [107, Chapter 5]) and to handle sparsity of solutions 
(see [3, 18, 19,50,52, 122] and [107, Section 3.3]), the systematic use of non-convex 
misfit and penalty functionals can be appropriate, in particular 0 < g < 1 for 
the norm power penalties ?. In the sequel, however, the focus in this section in 
combination with the noise model (2) is upon the functional 


@(x) = | F(x) — y* |? + aR), 1<p<oo, (11) 


with a convex penalty functional R to be minimized over D = D(F) N D(R) 
yielding minimizers . : 


Assumption 3. /. The operator F : D(F) © X — Y maps between reflexive 
Banach spaces X and Y with duals X* and Y*, respectively. 
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2. F is weakly sequentially closed, D(F) is convex and weakly closed, and the 
intersection D = D(F) N D(R) is nonempty. 

3. The functional R is convex and weakly sequentially lower semicontinuous. 

4. The functional R is stabilizing, which means that for every a > 0, c = 0, and 
for the exact right-hand side y of (1), the level sets 


Ma(c) := {x € D: ||F(x) — yl? + aR) < c} (12) 


are weakly sequentially pre-compact in the following sense: every infinite 
sequence {x,} in M,(c) has a subsequence, which is weakly convergent in X 
to some element from X. 

5. The operator F possesses for all xy) € D a one-sided directional derivative with 
Gdateaux derivative-like properties, i.e., for every Xx) € D there is a bounded 
linear operator F'(xo) : X — Y such that 


Jim, ~ (Fxg + tx —x0)) ~ Fao) = F (x0) — 0) 


is valid for all x € D(F). 

Under Assumption 3 existence and stability of regularized solutions x can be 
shown (see [60, §3]), i.e., the questions QI—Q2 above get a positive answer. In 
Banach spaces regularization errors are estimated by upper bounds of E (at. x), 
where F denotes an appropriate nonnegative error measure. Besides the norm 
deviation 


E (xé,xt) = |x —x'], (13) 


which is the standard error measure in Hilbert spaces, in Banach spaces and for 
convex functionals R with subdifferential 07, the Bregman distance 


E (x’,.x") = Dg (x8, x7) = R (x8) — R(x") — (€, x8 — xt) (14) 


at xt € Dg(R) C X andé € OR(x") C X* is frequently used as error measure, 
where the set Dg(R) := {x € D(R): dR(x) FM} represents the Bregman 
domain. An element x7 € D is called an R-minimizing solution to (1) if 


R (x!) = min{R(x): F(x) = y, x ED} <0. 


Such -minimizing solutions exist under Assumption 3 if (1) has a solution x € D. 
For given Qmax > 0 let x? denote an R-minimizing solution of (1). By setting 


p= 2? "lamax (1+ R(x‘) , (15) 


it holds xt € M (p) and there exists some dnax > 0 such that 


Omax 
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xi) € Mearmax (p) for all 0 = ) < bmax : 


Along the lines of [107, Section 3.2], [109, Chapters 3 and 4], and [3, 17, 58, 
60, 62, 65], there are presented in the following some results on the regularization 
theory for that setting. 

Under an a priori parameter choice w = a(5) > 0 satisfying 


Pp 
a(5) +0 and vu, as 6 > 0, 


a(6) 


a positive answer to question Q3 can be given and weak convergence eM Gt x! as 
6 — 0 (for subsequences in analogy to Proposition 1) of regularized solutions to 
R-minimizing solutions x‘ is shown. For stronger convergence results concerning 
the norm convergence, please refer, for example, to Proposition 3.32 in [107]. 

Taking into account the advantages of a posteriori choices of the regularization 
parameter a > 0, it is of interest to select such parameter choice rules ~@ = @ (5 es ) 
which obey the conditions 


Pp 
a(5,y°)>0 and ay >0 as 8-0. (16) 


Such study was performed for the sequential discrepancy principle which will be 
introduced in the following. The basis of this variety of discrepancy principle is, for 
prescribed 0 < gq < 1 and a > 0, a sequence 


Ag:={a;>0: a; =qiao, feQ (17) 
of regularization parameters and the specification of some constant t > 1. 
Definition 1 (sequential discrepancy principle). For fixed 5 > 0 and y’ € Y, it 


is said thata = a (5 ; y°) € A, is chosen according to the sequential discrepancy 
principle if 


[F (x8) -y*] <8 <[F (xh) — 9°. (18) 


Due to Assumption 3, the set Xin := {x € D: R(x) = Rmin} is nonempty for 
the value Rin := inf R(x) > Oand so is Yin := F(Xmin), where 
xe 


F(S):={peEY: p= F(x), x € S} 
is defined for subsets S C D(F). Moreover, there is an element Xin € Xmin such 
that dist (v°, Yin) = inf | F(a)= 9? | = | F(Xmin) — y° |. Using this notation, 
x€Xmin 


the Proposition 2 below still needs the following two definitions concerning the 
exact penalization veto and requirements on data compatibility, respectively. 
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Definition 2 (exact penalization veto). It is said that the exact penalization veto 
is satisfied for y € F(D) if, for all w > 0, any minimizer x? of the functional 


Do(x) := || F(x) — yl? + aR) 
over D, which is simultaneously an R-minimizing solution of (1), belongs to Xin. 


Definition 3 (compatible data). For y € F(D) and prescribed t > 1, it is said that 
there is data compatibility if there is some bax > 0 such that for all data y> € Y 
fulfilling (2), the condition 


16 <dist(y?, Yimin) forall O<68 <8mnax 


is satisfied. 


In the paper [3], there have been formulated sufficient conditions for fulfilling the 
exact penalization veto, which is mostly the case if p > 1 for the norm exponent 
in the misfit term of the Tikhonov functionals ® and ®o. Sufficient conditions 
for obtaining data compatibility can also be found in that paper. The following 
proposition is a direct consequence of the corresponding studies in [3] for an a- 
selection according to the sequential discrepancy principle. 


Proposition 2. Let for the exact right-hand side y € F(D) of Eq.(1) the exact 
penalization veto be satisfied and assume for prescribed t > 1 data compatibility. 
Then there is some bmax > 0 such that a = a (5, y) can be chosen according to 
the sequential discrepancy principle for all 0 < 5 < bmax. Moreover, this parameter 
choice satisfies condition (16), and consequently weak convergence si (58) al 
as 8 — 0 occurs in the sense of subsequences to R-minimizing solutions x* of (1) 


with the limit condition tne (465) =R (xt). 


The weak convergence of regularized solutions xe ( — x? in the sense of 


5,y8 ) 
subsequences from Proposition 2 is not worth too much. Even if the Radon-Riesz 


property, which ensures that x8 sy) > xt and R (<4, ») —> R(x‘) lead 
to ad (3.8) — xt as § — 0, allows amplifying this to norm convergence, this 


convergence can be arbitrarily slow for awkward solutions x‘. To get more, namely, 
a uniform error estimate for classes of solutions x‘ in the sense of convergence rates 
with an index function g determining the rate (see question Q4 above), additional 
requirements must be imposed on all elements x‘ from the class under consider- 
ation. In practice, one can restrict the considerations to rate functions g which are 
concave. The abovementioned requirements are always smoothness conditions. Pre- 
cisely, the R-minimizing solutions x‘ to be approximated by regularized solutions 
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must have a certain level of smoothness with respect to F’. For nonlinear forward 
operators F also the specific structure of nonlinearity influences the rate function @. 

With respect to operator properties, first the concept of a degree of nonlinearity 
from [58] is exploited: 


Definition 4 (degree of nonlinearity in Banach space). Let c; > 0, co > 0, and 
Cc, + co > 0. Then F is said to be nonlinear of degree (c;, cz) for the Bregman 
distance Dz (-, x") of R ata solution x! € Dg(R) C X of (1) with  € AR(x") C 
X* if there is a constant K > 0 such that 


JF) — F (xt) — Fe") (x —xt) J < K [FOF (eI De (21) 
(19) 
for all x € Mg,,,, (p). 


On the other hand, the solution smoothness of x* in combination with a well- 
defined degree of nonlinearity can be expressed in an efficient manner by variational 
inequalities 


(6, x7 — x) < BiDg (x,x1) + Bo |FQ)-—F (xt) |" forall x € Manax(0) 
(20) 


with some € € dR(x"), two multipliers 0 < B, < 1, fy > 0, and an exponent 
« > O for obtaining convergence rates. The subsequent theorem (for a proof see 
[65]) shows the utility of such variational inequalities for ensuring convergence rates 
in variational regularization (for more details in the case k = 1, see also [60] and 
[107, Section 3.2]). 

Theorem 4. For regularized solutions e minimizing ® from (11) with p > | under 
Assumption 3 and provided that there is an R-minimizing solution xt € Dg(R), the 
convergence rate 


E (xbq,2") = O(8*) as 530 (21) 


is valid for the Bregman distance (14) as error measure and for an a priori 
parameter choice a(6) = 6? if there exist an element — € OR (x‘) and constants 
0 < Bi <1, Bo = O such that the variational inequality (20) holds for some 
0 <x« < 1 andwith p from (15). 


This result which is based on Young’s inequality can immediately be extended 
to the situation 0 < «k < p < 1. Moreover, the situation k = p < 1 
characterizes the exact penalization case. For noisy data and « = p < 1, it holds 
Ds con x") = O (6?) as 6 > 0 for a regularization parameter ~@ = a which is 
fixed but sufficiently small (see [22]). An extension of such results to convergence 
rates of higher order is outlined in [97]. 
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To verify different situations for the exponent « > 0, the setting is restricted as 
follows: 


Assumption 5. Jn addition to Assumption 3, assume for an R-minimizing solution 

x" to (1): 

1. The operator F is Gateaux differentiable in x‘ with the Gdteaux derivative 
F’ (xf). 

2. The functional Ris Gateaux differentiable in x' with the Gdteaux derivative 


—E= R’ (x") € X*; hence, the subdifferential OR (x') = {&} is a singleton. 


The following proposition (see [65, Proposition 4.3]) shows that exponents « > 1 
in the variational inequality (20) under Assumption 5 in principle cannot occur. 


Proposition 3. Under the Assumption 5 the variational inequality (20) cannot hold 
with € = R! (x') # 0 and multipliers B,, By => 0 whenever k > 1. 


Now the following proposition will highlight the borderline case k = 1 and 
the cross connections between variational inequalities and source conditions for the 
Banach space setting. Moreover, in the next subsection, the interplay with (8) and 
generalized source condition can be discussed. 


Proposition 4. Under Assumption 5 the following two assertions hold: 


(a) The validity of a variational inequality 


(6.x? — x) < Bi Dg (x. x") + Bo || Fx) — F(x") forall x © Maa (P) 
(22) 
foré =F’ (x*) and two multipliers B,, Bz > 0 implies the source condition 


f= Fi(xt)"w,  weY*. (23) 


(b) Let F be nonlinear of degree (0, 1) for the Bregman distance Dg(-, x‘) of R at 
xt, ie, 


JF) — F (xt) — F(x!) (x1) < K Dg (2) 24 


holds for a constant K > O and all x € Mag,,,(p). Then the source 
condition (23) together with the smallness condition 


K ||wlly* < 1 (25) 


implies the validity of a variational inequality (22) with € = R’ (x7) and 
multipliers 0 < B, = K|\|wlly* <1, Bo = |lwlly* = 0. 
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Sufficient conditions for the validity of a variational inequality (20) with 
fractional exponents 0 < « < 1 are formulated in [58] based on the method of 
approximate source conditions using appropriate distance functions that measure 
the degree of violation of the source condition (23) for the solution xt. Assertions 
on convergence rates for that case can be made when the degree of nonlinearity is 
such that c, > 0 as the next proposition shows. 


Proposition 5. Under Assumption 5 let F be nonlinear of degree (c\, C2) with 
0<e < 1,0 <@ < 1c, +e < 1 for the Bregman distance Dg (-,x') of 
Ratx', ie, 


| FG) — F (x!) — F(x!) (es!) = K [ECF Us| Dy nt)? 
(26) 
holds for a constant K > 0 and all x € Myg,,,,(p). Then the source condition (23) 
immediately implies the validity of a variational inequality (20) with 


geo, (27) 
1-—© 


E=R’ (x"), and multipliers 0 < B, <1, Bo => 0. 


Some Specific Results for Hilbert Space Situations 


The abstract concepts of the last subsection for the Tikhonov regularization with 
quadratic functionals under Assumption 5, but with Assumption | in Hilbert spaces, 
will be illustrated next. For 


R(x) = |x — x* |? 


the 7-minimizing solutions and the classical x*-minimum norm solutions coincide. 
Moreover, it holds D = D(F) and for and D¢(x, x) the simple structure 


& =2 (x1 -x*) and Dg (&,x) = |X —x||? 


with Bregman domain Dg(R) = X. Then the source condition (23) attains the 
form (8) with A = F’ (x‘) /2. 

To focus on the distinguished character of the setting for Hilbert spaces X and 
Y, the Definition 4 will be specified as follows: 


Definition 5 (Degree of nonlinearity in Hilbert space). Let c; > 0, co > 0, and 
C1 + c2 > 0. Define F to be nonlinear of degree (ci, c2) at a solution xte D(F) 
of (1) if there is a constant K > 0 such that 
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JF) — F (6) — F(a) (62) < K JO) F (eI fat 
(28) 
for all x € Mg,,,, (p). 


Furthermore, in Hilbert spaces Hélder source conditions 
1(yt)* Bleyt a4 
g=(F (x*) F'(x')) vy, veX, (29) 


can be formulated that allow expressing a lower level of solution smoothness of x? 
for 0 < 7 < 1 compared to the case 7 = 1, where (29) is equivalent to 


& = F' (xt)" w, weY 


(cf. condition (8) in Theorem 2). For that situation of lower smoothness, the 
following theorem (see [65, Proposition 6.6]) complements the Theorem 2. 


Theorem 6. Under the Assumption 5 let the operator F mapping between the 
Hilbert spaces X and Y be nonlinear of degree (c,,c2) at x' with c, > 0 and 
let with R(x) = ||x — x*||? the element & = 2 (xi — x*) satisfy the Hélder source 
condition (29). Then the variational inequality (20) holds with exponent 


2nc1 2n 
1+n0—2c2)’ 14+7 


«= min \ 0<n<l, (30) 


for all x € Meama(e) and multipliers 0 < By, < 1, Bo > 0. Consequently, for 
regularized solutions xi 8) minimizing the functional ® from (5), the convergence 
rate 


E (x8,x") = O(5?) as 50 (31) 


holds for the norm distance (13) as error measure and for an a priori parameter 
choice 


cu(8) x 8°, 


For parameter identification problems in partial differential equations (cf., e.g., 
[9, 68]), which can be written as nonlinear operator equations (1) with implic- 
itly given forward operators F’, it is difficult to estimate the Taylor remainder 
| F(x)-F (x') — F' (x?) (x - x") | and the variational inequality approach may 
fail. However, if in addition to the Hilbert space X a densely defined subspace X 
with stronger norm is considered, then in many applications for all R > 0, there 
hold conditional stability estimates of the form 
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|x1 —22|| < K ||F(x1)— F(x)" if x1 Ee DLFYNX, |x lly < R G@ = 1,2) 

(32) 

with some 0 < « < | andaconstant K = K(R) > 0, which may depend on the 
radius R. 

Then along the lines of the paper, [31] the following theorem can be formulated. 


Theorem 7. Let X and Y be Hilbert spaces and let B : D(B) C X — X be 
an unbounded injective, positive definite, self-adjoint linear operator with domain 
X = D(B) dense in X. Furthermore, let C > 0 be a constant such that |x|] ¢ c= 
|| Bx|| = C ||x|| holds for x € X and X becomes a Hilbert space with norm || - ll ¢ 
stronger than the norm in X. For the nonlinear operator F : D(F) C X > Y, 
consider regularized solutions x as minimizers over D := D(F) 1 X of the 
functional 


2 
P(x) := || F(x) —y? | +o [lxI%. (33) 
Moreover, for all R > 0 let hold a conditional stability estimate of the form (32) 


with some 0 < « < 1 andaconstant K = K(R) > 0. Then for a solution x* € D 
of Eq. (1), the convergence rate 


E(xio.2") = O06) as 50 (34) 


is obtained with the norm distance (13) as error measure and for an a priori 
parameter choice c6* < a(8) < 8 for constants 0 < ¢ < t < @. 


The papers [28,65] show that the convergence rate result of Theorem 7 can be 
extended to Banach space situations. 


Further Convergence Rates Under Variational Inequalities 


Returning to the Banach space setting, in Theorem 4, convergence rates were drawn 
from the variational inequality (20) benefit. For the error measure F from (14) and 
taking into account that the penalty functional R is convex, this inequality can be 
reformulated as 


BE (x,x') < R(x) —R(x') + || Fx) — F(x’) forall xeM, 


for constants 0 < B < 1 and C > O, where M denotes an appropriate set 
which contains x‘ and all regularized solutions xé for sufficiently small 6 > 0. 
Such variational inequalities (also called variational range conditions or variational 
smoothness assumptions) are in the case of general nonnegative error measures E' a 
powerful tool for obtaining convergence rates in Tikhonov type regularization, and 
please refer to the overview in [42]. In the extended form 
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BE (x,x') < RXx)—- R(x") + Ce(|F(@)—F(x')|) forall xem, 
(35) 
with some concave index function g, such variational inequalities were intensively 
studied in [62]. Inequalities of this type combine assertions on the solution 
smoothness of x‘ with assertions on the structure of nonlinearity of F around x". In 
[17] it was shown that inequalities of the form (35) are a consequence of nonlinearity 
conditions 


|| F’ (x") (x — x')|| <0 (|F(@) —- F (x*)]) forall x eM, (36) 


with concave index functions o, where g depends on o, and both functions even 
coincide if x? satisfies the source condition 


& = F’ (xt)"w, weY*, (37) 


for some subgradient & € OR (x"). 


Theorem 8. Suppose that for y € F(D) the R-minimizing solution xt € D of 
Eq. (1) obeys for the nonnegative error measure E the variational inequality (35) 
with some set M © X, constants 0 < B <1, C > 0, anda concave index function 
y. Then, for the a priori parameter choice 


Pp 
a(6) = a& — 
g(5) 
and provided that the Tikhonov-regularized solutions satisfy the condition xe 8) € 
M for 0 <6 < bax, the convergence rate 


E(xi,x') =O) as 8 +0 (38) 


holds for arbitrarily chosen ag > 0 whenever the norm exponent p in (11) is taken 
from the interval 1 < p < cand for0 < ao < lifp =1. 


The same convergence rate result can also be derived from the variational inequal- 
ity (35) if the regularization parameter a is chosen according to the sequential 
discrepancy principle. For linear injective operators F and an £!-penalty term R, 
this was exploited in [21] (see also [4]) for obtaining convergence rates under 
conjectured sparsity constraints when the sparsity fails, but x‘ is in 2'!. Then the 
decay rate of solution components Ea — Oas k — ov influences the function g 
in (35). 
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3 Examples 


In this section, several examples of parameter identification problems in integral 
and differential equations will be presented in order to show how to apply the 
regularization theory outlined above to specific ill-posed and inverse problems. The 
examples refer either to nonlinear inverse problems, which can be formulated as 
operator equations (1) with forward operator F mapping from a Hilbert space X 
to a Hilbert space Y, or to linearizations of such problems, which then appear 
as linear operator equations. All discussed examples originally represent ill-posed 
problems in the sense that small data changes may lead to arbitrarily large errors 
in the solution. If the forward operator F is linear, then this phenomenon can be 
characterized by the fact that the range of the operator F is a nonclosed subspace 
in Y. For nonlinear F such simple characterization fails, but a local version of 
ill-posedness (see [64]) takes place in general. In order to make clear the cross 
connections to the theory, as in the previous sections, the unknown parameter 
functions are denoted by x, in particular the exact solution to (1) by x’, and the 
exact and noisy data are denoted by y and y®, respectively. For conciseness, this 
section restricts to six examples. More examples can be found in the corresponding 
references of this book. 


Example I ((Identification of coefficients in wave equations)). Let Q C R",n = 
1, 2,3, be a bounded domain with C >-boundary dQ. Consider 


6, t) = Au(é,t)+ x(Qu({,t), €€2,0<1t<T, 
u(é, 0) = a(), (E, 0) = b(¢), C € Q, (39) 
u(Et)=0, C€02, O<1<T. 


Here and henceforth 2 denotes the normal derivative. Fix initial values a and b 
such that 


da db 
ae HQ), be HQ), ay |a2x(o.r) = |a2x(0.7) = 0. (40) 


Then for any function x € W!°(Q), there exists a unique solution 
u(x) = u(x)(6,t) € C ([0, T]; H7(2))NC! ([0, T]; H?(Q)) NC? ([0, T]; H!(Q)) 


to (39) (see, e.g., [67]). 

The inverse problem here consists in the identification of the parameter function 
x = x(¢), € € Q, occurring in the hyperbolic partial differential equation based on 
noisy observations y° of time derivatives y of the state variable [u(x)](¢, ¢) on the 
boundary (¢, t) € dQ x (0, 7). In addition to (40), assume that 


T > min max |¢— ¢| (41) 
c/EQ CEQ 
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and 


|a(£)| > 0, ceEQ. (42) 


Moreover, set 
Un = {x € W'(Q) = |[xl|wicog < M} (43) 


for M > 0. 
In [67], it is proved that there exists a constant C = C(Q,7T,a,b, M) > O such 
that 


(44) 


0 
Ilx1 — xalle2@y SC | ay Wem) — u(X2)) 
t H1(92x(0,T)) 
for all x1, x. € Uy. 


The forward operator F is defined as a mapping from the space X¥ = L?(Q) to 
the space Y = H'(dQ x (0, T)) according to 


du(x) 


[FOG = 


|a2x(0.r) , (t)€d2x(0,T). 


This is a nonlinear operator mapping between the Hilbert spaces X and Y, and IIl- 
posedness of the corresponding operator equation can be indicated. However, the 
estimate (44) shows that this inverse problem possesses good stability properties if 
the set of admissible solutions is restricted suitably or if the regularization term is 
chosen in an appropriate manner. 

A Tikhonov regularization approach as outlined in the previous sections is useful. 
If the functional ® is chosen as 


(x) = | F(x) — y* |, tole — 371%, 


the theory applies. Alternatively, the penalty term can also be chosen by using a 
stronger norm. In this case, the functional ® is chosen as 


(x) = || FO) — yl, tall. 


where X = W!°(Q). 
Then by the conditional stability estimation (44), the convergence rate 


x2 = x" | 12(Q) 


= O(6) as 6>0 
is obtained with the choice a = 87. 


Example IIT (Determination of shapes of boundaries). Using polar coordinates 
(r,@), the shape of a boundary in R? is identified. For M > 0 and 0 < mo < 
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m, < 1, set 


d*x 


dkx 
gor = aor 27): k= 0, 1,2, 


Un,M = 5x =x(O)eE C?[0, 2x] : 


Ixlle2po.221 <M, IIxllefo2m) S71 


and 


Om, = {x € C7[0, 22]: x(0) = mo, 0< 0 <2zr}. 


Now with a function x € Un,,m, let Q(x) C IR? denote a domain being a subset of 
the unit circle, which is bounded by the curve y(x) = {€ = (7,0): r=x(@),0< 
@ < 2a}. Consider the Laplacian field in Q(x): 


Au =0 in Q(x), 


(45) 
Uvex) = 0, ulr = V, 
where y € C(I) is fixed and w > 0 does not vanish identically on I’. Then there 
exists a unique classical solution u(x) = u(x)(€) to (45). 

The inverse problem in this example is aimed at the identification of the interior 
subboundary y(x) from noisy data y> of y := bus) lr where I” is an arbitrary 
relatively open subset of T. 

In the paper [20], a uniqueness assertion was proved, namely, that it can be 
concluded, for x1,x2 € Unjm M Qmo, from the equality of two potential flux 
functions 


du(xi) _ du(x2) 


T’ 
ov ov 2 


that 
x1(0) = x2(8), 0<60<2n. 
Moreover, there exists a constant C = C(mo,m,, M, W) > 0 such that 


Cc 


du(x1) _ du(x2) 
dv dv 


(46) 


Ilx1 — X2|lcpo27] < 
log 


clr’) 


for all x1, x2 € Un) m OM Om. 
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The Banach spaces X and Y are fixed here as 
X=C[0,27], Y=c'r), 


and the forward operator is introduced by the assignment 


av Yr’ , 


Taking into account the intrinsic ill-posedness of this inverse problem, never- 
theless, it can be seen that the estimate (46) shows some weak, i.e., logarithmic, 
stability. This helps to overcome the ill-posedness here again if the admissible set is 
chosen suitably or the Tikhonov regularization is applied in an appropriate way. 

The theory of the preceding sections applies if the functional ® is chosen as 


(x) = | F(x) — yy], + eR) 


with #(x) as a convex penalty term or if the penalty term is equipped with some 
stronger norm leading to 


(x) = || F(x) — yl} + alx|%, 


where X = Qn, 9 Z and 


d*x d*x 
_ 2 4 = _ 
Z= 5x eEC’(0, 27]: SEO) = Fpem, k= 0.1. 


The conditional stability estimation (46) gives the convergence rate 


° 1 
Is -*"Lewan = (=) iis 


for the parameter choice a = 8. 
Similar inverse problems are discussed in the papers [14,58]. The regularization 
methods outlined above can be used to treat those inverse problems, too. 


Example III (Integral equation of the first kind with analytic kernel). Let D and D; 
be simple connected bounded domains in R* such that DM D, = @. Consider an 
integral equation of the first kind: 


x(¢) 
In— 5) 
This type of integral equation is derived in the context of models for nondestruc- 


tive testing (see [36]). In the original inverse problem, there is a nonlinear one. The 
integral equation (47), however, can be considered as a linearization of the original 


[F(@)|(n) = i. dt=y), ne Dt (47) 
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problem. It was shown in [36] that the linearized problem (47) is close to the original 
problem under some assumptions on the size of domain D. 

By DN D, = 9, the kernel ima is analytic inn € D, and¢ € D, so that (47) 
appears as a severely ill-posed linear operator equation. 

In the paper [30], it was proved that if there are two functions x;,x. € L?(D) 
such that the corresponding y1, y2 satisfy 


yin) = ya(n), ne Di, 


then it holds 


x1(0)= 2x20), CED. 


Moreover, the following conditional stability is proved: Fix g > 3 and 


2, 
ip {x EW (D): Ixlly2acpy S M}. 


Then there exists a constant C = C(q, M, D, D,) > 0 such that 


Ix lln2~D) S (48) 


|log Iv llary| 


for all x € Uy. 

The linear forward operator F maps here from the space X = L*(D) to the 
space Y = H'(D,). In spite of the original ill-posedness of the operator equation, 
the estimate (48) shows again logarithmic stability after appropriate restriction of 
the set of admissible solutions. 

Variational regularization with the Tikhonov functional 


O(x) = | F(x) —y* J + alle — 2" 
or alternatively with 
O(x) = | F(x) — y§] + allel 


for X¥ = We (D) allows the application of the general theory to that example. In 
particular, the conditional stability estimation (48) yields the convergence rate 


1 


I|x3 _ x" Lp) => O (on) as 6 > 0 


whenever the a priori choice w = 8? of the regularization parameter is used. 
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Example IV (Identification of wave sources). Let Q C R" be a bounded domain 
with C?-boundary dQ. Consider 


Fue t) = Au(E,t) +A(x(), x€Q,0<t<T, 


ar? 
u(6,0) = #(6,0)=0, Fe, (49) 
u(¢,t) = 0, Ce€dQ2, O0<t<T. 


Assume that 
AEC*™[0, 00), AO) £0, (50) 
and fix such A. Then for any function x € L7(Q), there exists a unique solution 
u(x) € C ([0, T]; H7(Q) N Hy (Q)) 9 C! ([0, T]; Hy (Q)) NC? ([0, T]; L7(Q)) . 
The inverse problem under consideration here is the identification of x = x(¢), 
¢ € Q, from observations y® of y := HOD | 50 (0,7): Corresponding uniqueness and 


conditional stability results can be found in [121]. 
Let 


3 
ee O<«<l, 
Kx A <K< 


Ale 


and let M > 0 be arbitrarily given. Set 


H*(Q), O<« 
X, = 
A5*(Q), 


where H**(Q), H?*(Q) denote the Sobolev spaces, and 
Unx = {x € Xx : Il || 242" (@) < M}. 
Furthermore, assume 


T >diamQ = sup |€-2'. (51) 


x,x/EQ 


Then, it is proved that there exists a constant C = C(Q,T,A,«) > 0 such that 


ok 
2K-- 


du(xi) _ u(x) 


1 
[x1 — aallex@) = CMT LG - 


L2(Qx(0,T)) 
for all x}, x2 € Uy. 
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The definition 


du(x) 


dv aQx(0,T) 


F(x) =yi:= 


of the forward operator F : X — Y is well defined for the Hilbert spaces X = 
L?(Q) and Y = L?(dQ x (0,T)). In contrast to Example I, where also a wave 
equation is under consideration, F appears here as a linear operator with nonclosed 
range. However, the estimate (52) shows that this inverse problem possesses even 
Holder type stability if we choose the admissible set suitably. 

With respect to the regularization methods from the previous sections, the 
functional ® can be chosen as 


(x) = | FO) — y* |p tarllx — x" 
or as 
(x) = | FO) — yj, tall, 


where ¥ = X,.. Then the conditional stability estimation (52) gives here the Holder 
convergence rate 


x2 - x" 2g) = O (87) as 6>0 


with the choice a = 82. 


Example V (Identification of potential in an elliptic equation). Let Q be a simply 
connected domain in R? with the C-boundary dQ. Consider the following problem 


(53) 


with functions x € L?(Q) and f € H2(aQ). 

Assuming that zero is not the Dirichlet eigenvalue of the Schrédinger operator 
A +x on the domain Q; it is known that there exists unique solution u € H!(Q) 
for this problem. Then the Dirichlet-to-Neumann map A, : H 2 (OQ) > A -3 (dQ) 
is defined as 


_ ou 


S| 5 54 
OV ag G4) 


Ax f 


where v is the unit outer normal with respect to dQ. 

The inverse problem under consideration here addresses the recovery of the not 
directly observable potential function x(¢), for ¢ € Q, from data y delivered by 
A... This is specified as follows: An infinite sequence {Vy }%_, of N-dimensional 
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subspaces Vy = span(fi, 2,..., fv) is considered generated by a basis { fj }72| 
in H'!(0Q), ie., 


[e.e) 
Vy C Vya1C H'(0Q2) ~— and |_) Vv is dense in H' (dQ). 
N=1 


In this context, it is assumed that the finite dimensional spaces Vy, N = 1,2,..., 


have the following properties: 


1. For any g € H'(0Q), there exists a gy € Vy and a function B(N), which 
satisfies limy—+o9 B(N) = 0, such that 


lg — gnll < BW )\lgll aca): (55) 


H2 (a2) 
2. There exists a constant C > 0, which is independent of g, such that 


lev lk ey = Cell gh ey: (56) 


The following result is proved in [29]: Suppose that x; € H*(Q), 7 = 1,2, with 
s> 3, satisfy 


Ix; las@y < M 


for some constant M > 0. Then there exists a constant C > 0, which depends on 
M, such that 


Il41 — Xallz2@Qy S Co (Ax, — Axollyy + BY) (57) 


for N large enough and |/Ax, — Ax,||vy small enough. Precisely, it is here w(t) = 


(4 T ) with some 0 < y < | taking into account that 
t 


Ax, — Avallyy = sup | (Ax, — Axa), 6) |, 
geVy Ill 1 =1 
H2 (32) 


where (-, -) is the dual pairing between H-? (dQ) to H2(dQ). 
Here the forward operator F is defined as 


F(q) = Alyy. 


This is a nonlinear operator mapping from the space X = L*(Q) into the space 
Yel (202), H? (32), which represents the space of bounded linear operators 


mapping between L?(0Q) and H 2 (092). Moreover, consider the restriction Yy = 
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Y |vy of Y generated by the subspace Vy. The original problem of finding x from 
A, data is ill-posed, but even without the uniqueness of the inverse problem, the 
estimate (57) shows some stability behavior of logarithmic type under the associated 
restrictions on the expected solution. Again for the Tikhonov regularization with 
functionals 


(x) = | F@)—y* |}, + alle — 27 
or 
(x) = |FO)—y* |}, + allel, 


where X = H°* with s > 3, the theory of sections “Tikhonov Regularization 
in Hilbert Spaces with Quadratic Misfit and Penalty Terms” and “Variational 
Regularization in Banach Spaces with Convex Penalty Term” is applicable. From 
the latter section, with the conditional stability estimation (57), the convergence rate 


x2 - #" | ates = O((log(1/5))”) as 56-0 


can be derived for the parameter choice a = 8”. 


Example VI (Inverse problems for autoconvolution equations). For the space X = 
Y = L7(0, 1) of quadratically integrable real functions defined on the unit interval, 
consider as first variant of this example the autoconvolution equation x * x = y, 
where with reference to Eq. (1), 


Ss 


[F(x)](s) := [x —t)x(t)dt, O<s<1 (58) 
0 
is the corresponding forward operator with D(F) = X. This operator equation 


of quadratic type occurs in physics of spectra, in optics, and in stochastics, often 
as part of a more complex task (see, e.g., [11, 70, 108]). A series of studies on 
deautoconvolution and regularization have been published for the setting (58); see, 
for example, [34, 100]. Some first basic mathematical analysis of the autoconvolu- 
tion equation can already be found in the paper [48]. Moreover, a regularization 
approach for general quadratic operator equations was suggested in the recent 
paper [43]. Because of their weak nonlinearity, deautoconvolution problems are 
not seen as difficult, and hence, little attention is paid to them wrongly. However, 
there is a deficit in convergence rates for regularized solutions x obtained by the 
classical form of Tikhonov regularization in Hilbert spaces as minimizers of the 
functional (5). It can be shown that Assumption | applies and that the inequality (7) 
from Theorem 2 is satisfied with L = 2 and for arbitrary large radii p, even as an 
equation 
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[FF (xt) —F (xt) 2), = [ety 


where 


[F’(x)h] (s) = 2 f x6 —t)h(t)dt, O<s<1, he L?(0,1), 


0 


characterizes the Fréchet derivative of F from (58) at the point xt. Note that it is 
very specific phenomenon here that the nonlinear operator F' is not compact, but 
F'(x") is a compact linear operator mapping in L?(0, 1). One consequence of this 
specific interplay between F and its derivative for the deautoconvolution problem 
from (58), which is locally ill-posed everywhere, is the fact that the classical 
convergence rate theory developed for the Tikhonov regularization of nonlinear ill- 
posed problems reaches its limits if standard source condition 


xt —x* = F’ (xt)" w, wey, 


fails. Please refer to [24] for details. On the other hand, convergence rate results 
based on Hélder source conditions with small Hélder exponent and logarithmic 
source conditions or on the method of approximate source conditions (cf. [58]) are 
not applicable since qualified nonlinearity conditions like (36) cannot be proven 
according to current knowledge. 

For a function x : R — R with support on [0, 1], the autoconvolution x * x 
is a function with support on [0,2]. Hence, the strength of ill-posedness for the 
deautoconvolution problem according to the forward operator (58) can be reduced 
if observations of [F'(x)](s) for all 0 < s < 2 are taken into account (full data 
case). Please refer to [46] for details and numerical case studies. In a second variant 
of this example with applications in laser optics, the full data case is exploited, 
but for complex functions and with an additional kernel. This variety considers 
a generalized autoconvolution equation motivated by problems of ultrashort laser 
pulse characterization arising in the context of the self-diffraction SPIDER method, 
and the reader is referred to the recent paper [47] for physical details and the 
experimental setting of this problem. In this variant, the focus is on a kernel- 
based, complex-valued, and full data analog to (58). Take into account Lz-spaces 
of quadratically integrable complex-valued functions over finite real intervals set for 
the corresponding normed spaces X = L2.(0, 1) and Y = LAO, 1) and consider 
for the associated operator equation (1) the nonlinear forward operator 


fkts,A)x —t)x(t)dt if0<s<1 


[FO)M(s) = 1°, (59) 
f k(s,0)x(s —t)x(t)dt ifl<s <2 
\ 


s— 
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mapping from LO; 1) to L200, 2) with domain D(F) = L2.(0, 1). Every function 
x € L2(0,1) can be represented as x(t) = A(t)e®, 0 < ¢ < 1, with the 
nonnegative amplitude (modulus) function A = |x| and the phase function ¢ : 
[0, 1] — R. For the SPIDER technology, in particular, the phase function is to be 
determined from noisy observations of the complex function y, whereas information 
about the amplitude function can be verified by alternative measurements. In [47] 
some mathematical studies and a regularization approach for this specific problem 
have been presented, and further analytic investigations for the specific case of a 
constant kernel k can be found in [24]. 


4 Conclusion 


This chapter has presented some theoretic results including convergence and 
convergence rates assertions on direct regularization methods for nonlinear inverse 
problems formulated in the setting of infinite dimensional Hilbert or Banach spaces. 
The inverse problems can be written as ill-posed nonlinear operator equations 
with the consequence that their solutions tend to be unstable with respect to 
data perturbations. To overcome that drawback, regularization methods use stable 
auxiliary problems, which are close to the original inverse problem. A regularization 
parameter controls the trade-off between approximation and stability. For direct 
regularization methods, the auxiliary problems are mostly minimization problems 
in abstract spaces, where a weighted sum of a residual term that expresses the data 
misfit and a stabilizing penalty term expressing expected solution properties has 
to be minimized. In this context the regularization parameter controls the relative 
weight of both terms. Furthermore, six examples are given that show the wide 
range of applicability for such regularization methods in the light of specific inverse 
problems. More than 120 references at the end of this chapter survey the relevant 
literature in this field. 
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1 Introduction 


Today imaging is rapidly improving by increased specificity and sensitivity of 
measurement devices. However, even more diagnostic information can be gained 
by combination of data recorded with different imaging systems. 

In particular in medicine, information of different modalities is used for diag- 
nosis. From the various imaging technologies used in medicine, we mention 
exemplary positron emission tomography (PET), single photon emission computed 
tomography (SPECT), magnetic resonance imaging (MRI), magnetic resonance 
spectroscopy (MRS), X-ray, and ultrasound. Soft tissue can be well visualized in 
magnetic resonance scans, while bone structures are more easily discernible by X- 
ray imaging. 

Image registration is an appropriate tool to align the information gained from 
different modalities. Thereby, it is necessary to use similarity measures that are able 
to compare images of different modalities, such that in a post-processing step the 
data can be fused and relevant information can be aligned. 

The main challenge for computer-assisted comparison of images from different 
modalities is to define an appropriate distance measure between the images from 
different modalities. 

Similarity measures of images can be categorized as follows: 


1. Pixel-wise comparison of intensities. 

2. A morphological measure defines the distance between images by the distance 
between their level sets. 

3. Measures based on the image’s gray value distributions. 


In the following, we review distance measures for images according to the above 
catalog. 

We use the notation 92 for the squared domain (0, 1)”. Images are simultaneously 
considered as matrices or functions on §2: A discrete image is an N x N-matrix 
U e€ {0,..., VS alae Each of the entries of the matrix represents an intensity 
value at a pixel. Therewith is associated a piecewise constant function 


un) = oy, Do, xa, (0), (1) 


where 


noth els oh 
Se el ee eae 
NN NN 


and Xj is the characteristic function of §2;. In the context of image processing, 


U! denotes the pixel intensity at the pixel x qi. A continuous image is a function 
u: Q2Q>R. 
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We emphasize that the measures for comparing images, presented below, can be 
applied in a straightforward way to higher-dimensional domains, for example, voxel 
data. However, here, for the sake of simplicity of notation and readability, we restrict 
attention to a two-dimensional squared domain (2. Even more, we restrict attention 
to intensity data and do not consider vector-valued data, such as color images or 
tensor data. By this restriction, we exclude, for instance, feature-based intensity 
measures. 


2 Distance Measures 


In the following, we review distance measures for comparing discrete and continu- 
ous images. We review the standard and a morphological distance measure; both of 
them are deterministic. Moreover, based on the idea to consider images as random 
variable, we consider in the last two subsections two statistical approaches. 


Deterministic Pixel Measure 


The most widely used distance measures for discrete and continuous images are the 
1, L? distance measures, respectively, in particular p = 2; see, for instance, the 
chapter » Linear Inverse Problems in this handbook. There, two discrete images Uj 
and U2 are similar, if 


1 
N N ij ii|?\? 
|U1 — Vall, = Sy ai e ie l<p<o, 


E ] P=, 


respectively, is small. Two continuous images u;, u2: $2 —> R are similar if 


1 
l|ur — wel), = (f alur(x) — u2(x)|?, dx)? 1 < p < 00, 
||u2 — Urloo = ESS sup, y [U1 (x) — u2(Xx)|, P =v, 


is small. Here, ess sup denotes the essential supremum. 
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Qo.5(U4) 


Q4 (U4) 


Uy morphology of Uy ,U5 


Fig. 1 The gray values of the images are completely different, but the images , uz have the same 
morphology 


Morphological Measures 


In this subsection, we consider continuous images u; : $2 —> [0, 255], i = 1, 2. 
uy and uz are morphologically equivalent (Fig. 1), if there exists a one-to-one gray 
value transformation B : [0, 255] > [0, 255], such that 


Bou =u. 
Level sets of a continuous function u are defined as 
Q,(u) = {x € Q: u(x) = th. 


The level sets 2p (u) := {2;(u): t € [0, 255]} form the objects of an image that 
remain invariant under gray value transformations. The normal field (Gauss map) is 
given by the normals to the level lines and can be written as 


niu): Q > R? 


0 if Vu(x) =0 
Vu(x) 


XxX ke 1 
Tvuca Se: 


Droske and Rumpf [7] consider images as similar, if intensity changes occur at the 
same locations. Therefore, they compare the normal fields of the images with the 
similarity measure 


S_(us, 42) = [ saa) ava @re (2) 


where they choose the function g : R? x R* — R > 0 appropriately. The vectors 
n(u,)(x), m(uz)(x) form an angle that is minimal if the images are morphologically 
equivalent. Therefore, an appropriate choice of the function g is an increasing 
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function of the minimal angle between v,, v2, and vj, —v2. For instance, setting 
g to be the cross or the negative dot product, we obtain 


Sx (ui, U2) = 3 fo |m@ur)() x mn)(x)? dx 
So(ui,u2) = 3 fg A — m(ui)(x) -mun)(x))° dx. 


(The vectors n have to be embedded in R? in order to calculate the cross product.) 


Example 1. Consider the following scaled images u; : [0, 1]* — [0, 1], 
u(x) = X1xX2, U(x) = 1—x1x2, w3(x) = (1 — x1) x2, 


with gradients 


Vii (x) = (7), Vuo(x) = (="). Vin(x) = a) 


With g(u, v) := $|uiv2— uv; |, the functional S, defined in (2) attains the following 
values for the particular images: 


So(uj, U2) = | — xx, + xX2x)| dx = 0 


1 
ip | 

So(u2, U3) = 5 fo |[xox1 + X2x1| dx = 3 

Se(u3, U1) = 5 fo | — 2x1 — (= x1) x2| dx = i 


The similarity measure indicates that uw; and uz are morphologically identical. 


The normalized gradient field is set valued in regions where the function 
is constant. Therefore, the numerical evaluation of the gradient field is highly 
unstable. To overcome this drawback, Haber and Modersitzki [15] suggested to use 
regularized normal gradient fields: 


n(u): Q> R’ 2 
Ux 
Xe [Wacol 


where ||v||. := /v7v + é2 for every v € R¢. The parameter € is connected to 
the estimated noise level in the image. In regions where € is much larger than the 
gradient, the regularized normalized fields n.(u) are almost zero and therefore do 
not have a significant effect of the measures Sx or So, respectively. However, in 
regions where € is much smaller than the gradients, the regularized normal fields 
are close to the non-regularized ones (Fig. 2). 
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Fig. 2 Top: images uy, uz, U3. Bottom: n(u), n(uz), n(us3) 


Statistical Distance Measures 


Several distance measures for pairs of images can be motivated from statistics by 
considering the images as random variables. In the following, we analyze discrete 
images from a statistical point of view. For this purpose, we need some elementary 
statistical definitions. Applications of the following measures are mentioned in 


section “Morphological Measures”: 


Correlation Coefficient : 


N N 
- 1 F —s 
U := W2 ) U" and Var(U) = ) (U!—U) 
ij=l ij=l 


denote the mean intensity and variance of the discrete image U. 


Cov(U,, U2) = 3 > (ui = 0) (us ~ Os) 


i=1j=1 


denotes the covariance of two images U; and U3, and the correlation coefficient is 


defined by 
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Cov(U,, U2) 


p(U,, U2) = Wari) Var(Usy 


The correlation coefficient is a measure of linear dependence of two images. The 
range of the correlation coefficient is [—1, 1], and if p(U;, U2) is close to one, then 
it indicates that U; and U2 are linearly dependent. 


Correlation Ratio: In statistics, the correlation ratio is used to measure the 
relationship between the statistical dispersion within individual categories and the 
dispersion across the whole population. The correlation ratio is defined by 


Var(E(U2|U1)) 


n(U2|U\) = Var(U>) 


where E(U2|U;) is the conditional expectation of U2 subject to U;. 
To put this into the context of image comparison, let 


2.(Ui) = {GDIU! = 4} 


be the discrete level set of intensity t € {0,..., 255}. Then the expected value of U2 
on the f-th level set of U; is given by 


1 *¢ 
E(U2|U; = t) := ~—~———— Us, 
IN = = Fe GD a 


where #(2,;(U;)) denotes the number of pixels in U, with gray value t. Moreover, 
the according conditional variance is defined by 


1 ‘i oy? 


The function 


H(U;) : {0,...,255} > N 
t b> #(Q,(U;)) 


is called the discrete histogram of U,. 

The correlation ratio is nonsymmetric, that is, n(Y |X) 4 n(X|Y), and takes values 
in [0, 1]. Itis a measure of (non linear dependence between two images. If U; = U, 
then the correlation ratio is maximal. 


Variance of Intensity Ratio, Ratio Image Uniformity: This measure is based on 
the definition of similarity that two images are similar, if the factor R! (U,, Ur) = 
U,'/U, has a small variance. The ratio image uniformity (or normalized variance 
of the intensity ratio) can be calculated by 
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4 4 5 6 6 [3] 
3 3/4 4 BS 8 
2 2/3 fae 
12 2; 
11/2 238 
11128 

U; U3 


Fig. 3 Images for Examples 2 and 6. Note that there is a dependence between U, and Uz: U2 ~ 
11-(U,)3 


Table 1 Comparison of the different pixel-based similarity measures. The images U;, U2 are 
related in a nonlinear way; this is reflected in a correlation ratio of 1. We see that the variance of 
intensity ratio is not symmetric and not significant to make a statement on a correlation between 


the images 
Ui, Us __| U2, Us 
Correlation coefficient —0.98 —0.98 —0.14 
Correlation ratio 1.00 1.00 0.64 
Variance of intensity ratio 1.91 2.87 0.83 


RIU(U;, Ur) = _ 


It is not symmetric. 


Example 2. Consider the discrete images U;, U2, and U3 in Fig. 3. Table | shows a 
comparison of the different similarity measures. The variance of the intensity ratio is 
insignificant and therefore cannot be used to determine similarities. The correlation 
ratio is maximal for the pairing U;, U2, and in fact there is a functional dependence 
of the intensity values of U; and U2. However, the dependence of the intensity values 
of U; and U2 is nonlinear; hence, the absolute value of the correlation coefficient 
(measure of linear dependence) is close to one, but not identical to one. 


Statistical Distance Measures (Density Based) 


In general, two images of the same object but of different modalities have a large 
L?, 1? distance. Hence, the idea is to apply statistical tools that consider images 
as similar if there is some statistical dependence. Statistical similarity measures are 
able to compare probability density functions. Hence, we first need to relate images 
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to density functions. Therefore, we consider an image as a random variable. The 
basic terminology of random variables is as follows: 


Definition 1. A continuous random variable is a real-valued function X : 2° + R 
defined on the sample space Q°. For a sample x, X(x) is called observation. 


Remark I (Images as Random Variables). When we consider an image u : S2 > R 
as a continuous random variable, the sample space is §2. For a sample x € £2, the 
observation u(x) is the intensity of u at x. 


Regarding the intensity values of an image as an observation of a random 
process allows us to compare images via their intrinsic probability densities. Since 
the density cannot be calculated directly, it has to be estimated. This is outlined 
in section “Density Estimation”. There exists a variety of distance measures for 
probability densities (see, for instance, [31]). In particular, we review f -divergences 
in section “Csiszar-Divergences (f-Divergences)” and explain how to use the f- 
information as an image similarity measure in section “ f-Information”. 


Density Estimation 
This section reviews the problem of density estimation, which is the construction of 


an estimate of the density function from the observed data. 


Definition 2. Let X : Q° — R be a random variable, that is, a function mapping 
the (measurable) sample space 2° of a random process to the real numbers. 


The cumulated probability density function of X is defined by 


P(t) := meas{x : X(x) <t}teER. 


meas(Q%) 


The probability density function p is the derivative of P. 
The joint cumulated probability density function of two random variables X,, X2 
is defined by 


P(t, t) = meas{(x1, 2): X1(x1) < th, X2(%2) < bh} ti, ER. 


meas(Q*°)? 


The joint probability density function p satisfies 
K ty ty 
P (ti, t2) = i i D(s1, S2)dsy ds>. 
0 JO 


Remark 2. When we consider an image u : (2 + R arandom variable with sample 
space §2, we write p(u)(t) for the probability density function of the image u. For 
the joint probability of two images uw; and u2, we write p(u1, u2)(t1, t2) to emphasize, 
as above, that the images are considered as random variables. 
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The terminology of Definition 2 is clarified by the following one-dimensional 
example: 


Example 3. Let Q := [0, 1] and 


u: Q > [0,255] 
x > 255x?, 


The cumulated probability density function 
P: [0, 255] > [0, 1] 
is obtained by integration: 


i 255 i 
P(t) := meas{x :255 x” <1} = meas¢x:x < ,/——} = i <j dx =,/—. 
255 0 255 


The probability density function of u is given by the derivative of P, which is 


1 1 
2/255 Vt 
In image processing, it is common to view the discrete image U (or uy as in (1)) as 
an approximation of an image u. We aim for the probability density function of u, 
which is approximated via kernel density estimation using the available information 
of u, which is U. A kernel histogram is the normalized probability density function 
according to the discretized image U, where for each pixel a kernel function (see 


(3)) is superimposed. Kernel functions depend on a parameter, which can be used to 
control the smoothness of the kernel histogram. 


PUu)(t) = 


We first give a general definition of kernel density estimation: 


Definition 3 (Kernel Density Estimation). Let ¢,, t2, ..., ty be a sample of 
M independent observations from a measurable real random variable X with 
probability density function p. A kernel density approximation at t is given by 


1 M 
Po(t) = WT Quins Kot =), t€ 0,255) 


where k, is a kernel function with bandwidth o. p, is called kernel density 
approximation with parameter o. 


Let f1,f2,..., tv and s1,52,...,S4 be samples of M independent observations 
from measurable real random variables X,, X2 with joint probability density 
function p; then a joint kernel density approximation of p is given by 
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Fig. 4 Density estimate for 
different parameters o 


. i aig 
Po(s,t) = u D4 K,(s — s;,t —t;), 


where K,(s, t) is a two-dimensional kernel function. 
Remark 3 (Kernel Density Estimation of an Image, Fig. 4). Let u be a continuous 
image, which is identified with a random variable. Moreover, let U be N x N 


samples of u. In analogy to Definition 3, we denote the kernel density estimation 
based on the discrete image U, by 


1 N 33 
— = y 
Pot) = N2 ) ipa Fol! U") 


and remark that for wy as in (1) 


potuny(d) i= f holt —uy(a)de= 5 Dy jhe), ) 


The joint kernel density of two images u;, v2 with observations U; and U} is 
given by 


x 1 N ij ij 
Po(s,t) = NW? De Ky (s _ ule _ u}) , 
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where K,(s, t) = kg(s)ko(t) is the two-dimensional kernel function. Moreover, we 
remark that for wu, .y, U2,n 


Bo(tv.tan V(t) = ff Ka(s— ay @)ot ~uay(x)) ax 
Q 


is = i (s—uj!,t—-Uy). 


In the following, we review particular kernel functions and show that standard 
histograms are kernel density estimations. 


Example 4. Assume that uj : (2 — [0, 255], i = 1, 2 are continuous images, 
with discrete approximations u;,y as in (1): 


¢ We use the joint density kernel K,(s, t) := ko(s)ko(t), where kg is the 
normalized Gaussian kernel of variance o. Then fori = 1, 2, the estimates 
for the marginal densities are given by 


1 =(uj —ty 
Poltsn)(t) = ff olan (s) =) dx = —— ff exp (VXI) ay, 


and the joint density approximation reads as follows: 


Po(s,t) : = JQ Ko((ui(x), u2(x)) — (s, t)) dx ; 
= 1, Joexp (ea ) exp (eg ) ae 
¢ Histograms: Assume that U only takes values in 0, 1,...,255. When we choose 


the characteristic function X|-5,), witha = 5 


density estimate 


as kernel function, we obtain the 


II 


Pyo(t) is X(-o.0] (u(x) = t) dx 
meas{x :f —o <u(x) <t+o} 


size of pixel x number of pixels with value |t + o | 


II 


Hence, p,,, corresponds with the histogram of the discrete image. 


Example 5. We return to Example 3. The domain §2 = [0, 1] is partitioned into N 
equidistant pieces. Let 
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+ + + 
3 = = ss = 
> =o > os > 
x x x 


Fig. 5 Original u and discretized versions uy and uy! 


a b c d 


Fig. 6 (a) Density from the original image wu. (b) Density estimation with Gaussian-kernel based 
onuy(N = 10), with o = 0.07, (ec, d) normalized histogram, based on uy’, witho = 0.05, 0.1 


Moreover, we consider the piecewise function uy’ represented in Fig.5. The 
density according to u, denoted by p(u), and the kernel density estimates of uy 
and uy! are represented in Fig. 6. They resemble the actual density very well. 


Csiszar Divergences (f-Divergences) 

The concept of f-divergences has been introduced by Csiszar in [5] as a general- 
ization of Kullback’s /-divergence and Rényi’s /-divergence and at the same time 
by Ali and Silvey [1]. In probability calculus, f-divergences are used to measure 
the distances between probability densities. 


Definition 4. Set Fo := {f : [0, co) > RU {oo} : f is convex in [0, oo), 
continuous at 0, and satisfies f(1) = 0} and 


Vat = pe Lie): p=0.f poa=i}. 
R 
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Let g1, g2 € Vpar be probability density functions. The f-divergence between g1, 
&2 is given by 
Df ; Vat x Vpat > [0, oo) 


4 
(91.82) > fa OS (28) at © 
Remark 4. 


¢ In (4), the integrand at positions t where g2(¢) = 0 is understood in the following 


sense: 
of (22) = lim (ir (2°) _teR. 


¢ In general, f-divergences are not symmetric, unless there exists some number c 
such that the generating f satisfies f(x) = x f (4) + c(x — 1). 


Examples for f-Divergences: We list several f-divergences that have been used 
in literature (see [6, 12] and references therein). 


The Kullback-Leibler divergence is the f-divergence with f(x) = x log(x) 


Dy (g1, 82) = [ sito Ga dt. 


g(t) 


Jensen-Shannon divergence is the symmetric Kullback-Leibler divergence: 


D (81, 82) = [ (errs (£2) + ex(t)log ( 27) ae 


f(x) 


x°-Divergences: These divergences are generated by 
f(x) = |x-1), s € [1,00) 


and have the form 
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f (x) 


gi(t) 
g2(t) 


Df (g1, 2) = [sx 


= 1 ~ / 87 “(igi (t) — g2@) Ide. 


The x!-divergence is a metric. The most widely used out of this family of x° 
divergences is the x?-divergence (Pearson). 


Dichotomy Class Divergences: The generating function of this class is given by 


x —1—In(x) for s = 0, 
f(x) = Way (8 + 1—s—x°‘) fors € R\{0, 1}, 
1—x+x In(x) fors =1. 


The parameter s = 5 provides a distance, namely, the Hellinger metric 


D ¢(81, 82) = 2f (Vai - Ve) dt. 


f (x) 
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Matsushita’s Divergences: The elements of this class, which is generated by the 
function 


f(x) = [l-x|F, O<s <1, 


are prototypes of metric divergences. The distance is given by 


d(g1, 82) = (Dy(g1, 82))” 


where 


nl 
s 


i- (22) 
gi(t) 


Puri-Vincze Divergences: This class is generated by the functions 


D (81,82) = [sw 


For s = 2, we obtain the triangular divergence 


(ga(t) — gi) 5, 


Ie | eae). 


Divergences of Arimoto Type: Generated by the functions 


(+5): —2!-1(1 + x)) for s € (0,00)\{1} 
IO) =% G+ x) In(2) + x In(x) -(1 + x) In. + x) fors = 1 
5|1 — x| for s = ©. 
For s = oo, the divergence is proportional to the yx! divergence. For s € 


(0, co)\{1}, we obtain 
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f (x) 


D (81,82) = i (y gi(t) + a(t) — 2° (gi(t) + nt) dt. 


Moreover, this class provides the distances 


d(gi, 0) = (Dp(gi,go))m™"} for s € (0,00). 


f -Information 


In the following, we review the /-information for measuring the distance between 
probability densities. The most important f-information measure is the mutual 
information. 

The notion of information gain induced by simultaneously observing two 
probability measures compared to their separate observations is tightly related to 
divergences. It results from quantifying the information content of the joint measure 
in comparison with the product measure. 

This motivation leads to the following definition. 


Definition 5 ( f-Information for Images). For f € Fp (see Definition 4), we 
define the f-information of u,, uw. € L° (2) by 


T¢ (ui, U2) = Dy (p(u1) pa), PU, ¥2)), 
where the p(u;) is the probability density of u;, as introduced in section “Density 


Estimation.” 
Additionally, we define the f-entropy of an image u; by 


Hf (u1) = T7(u, uy). 


In analogy to independent probability densities, we call two images wu, uz indepen- 
dent if there is no information gain, that is, 


P(uy, U2) = p(ur) p(w). 
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Remark 5. The f-information has the following properties: 

¢ Symmetry: J ¢(u;, u2) = If (u2, 1). 

* Bounds: 0 < Jf (uj, uz) < min{H (1), Hp (u2)}. 

¢ Ip¢(u, uz) = Oif and only if u), v2 are mutually independent. 

The definition of f-information does not make assumptions on the relationship 
between the image intensities (see [38] for discussion). It does neither assume a 
linear nor a functional correlation but only a predictable relationship. For more 
information on f-information, see [36]. 

Example 6. The most famous examples of f-informations are the following: 


Mutual/Shannon Information: For f(x) = x Inx, we obtain 


= P(uy, Uz) (t1, t2) 
Can Lhe ees (stow ruaen) ae 


with Shannon entropy 


1 
HW) = [ pac in(— dt, 


joint entropy 
H(i) =~ ff poarany(er ta) n (plans ae) rst) dt a, 
conditional entropy 
Hy(ushun) =f plan)(e) Harlan =) de 


and relative entropy (the Kullback-Leibler divergence) 


Bae) = [rein (A) 


The relative entropy is not symmetric. Maes et al. [25] and Studholme et al. [35] both 
suggested the use of joint entropy for multimodal image registration. Maes et al. 
demonstrate the robustness of registration, using mutual information with respect to 
partial overlap and image degradation, such as noise and intensity inhomogeneities. 


Hellinger Information: For f(x) = 2x — 2 —4./x (see also Dichotomy Class in 
section ““Csiszar-Divergences ( f-Divergences)”), we obtain 
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Tp(u1, to) = i i (Vrtu.wytt.2) — Vou) pay) dr dts, 


with Hellinger entropy 


H;(w) =2 (: ae (nto) at) | 


Both are bounded from above by 2. 


For measuring the distance between discrete images U; and Us, it is common 
to map the images via kernel estimation to continuous estimates of their intensity 
value densities po(uj.y), where po(uj.v) is as defined in (3). The difference 
between images is then measured via the distance between the associated estimated 
probability densities. 


Example 7. For U;,i = 1,2,3 as in Fig. 3, let u;,y be the corresponding piecewise 
constant functions. Note that U; and U; are somehow related. In other words, they 
are highly dependent on each other, so we can expect a low information value. 
Comparing the images point-wise with least squares shows a higher similarity value 
for U> and U3 than for U; and U3. 


For the ease of presentation, we work with histograms. Recall that the estimated 
probability function p,(uj,y) is equal to the normalized histogram of U;. The 
histograms (connected to the marginal density densities) are given by 


i 2S A 66 9-8 8 16 
HU) |6 769 3 4100 0 
HU)|1 040309 67 6 
HUs;)|3 2 53 2405 6 6 


In order to calculate the information measures, we calculate the joint histograms of 
U;, U2, U3, that is, JH(U;, Ur) : (s, t) > number of pixels such that U’ = s and 
U:! = t (see Tables 2 and 3). 

The entries in the joint histogram of U), U2 are located along a diagonal, whereas 
the entries of the other two joint histograms are spread all over. Hence, we can 
observe the dependence of po(uiw), Po(U2,n) already by looking at the joint 
histogram. Next, we calculate the Hellinger and the mutual information. For the 
f -entropies, we obtain 


U, UW, Us; 
Mutual entropy 1.91 1.91 2.13 
Hellinger entropy | 1.17 1.17 1.30 
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Table 2 Joint histograms of U2, U3 and U3, U;. The entries are disperse; this will be reflected in 
a lower f -information as in the case for U;, Uz 


S 

JH(U;,Us) |1 23456789 10[= 
1 1 
2 0 
3 4 
4 0 
5 3 
6 0 
7 9 
8 6 
9 7 
1016 6 


Table 3 Joint histograms of Uz, U3 and U3, U;. The entries are disperse; this will be reflected in 
a lower f -information as in the case for U,, Uz 


Sy SI 

JH (Up,U3)11 23.45 67.8 910|—E  JH(Us,Uy) 678 910|r 
1 117 13 1 22216 
2 2 2 2 1 2 7 
3 { 2 2 Is 3 1 11/6 
4 2 1 3 4 1 12 2I9 
5 1 1 |2 5 1 3 
6 1 did [4 6 1/4 
7 0 Fd 1 14 
8 1 2215 8 0 
g|1 21 2(6 9 0 
10 1 21 2/6 10 0 
A) ARONA RONG DOMOMGNNG H(U;)|8 253240566 


and for the f-information measures: 


(U;,U2) (U2, U3) (U3, Ui) 
Mutual information 1.91 0.74 0.74 
Hellinger information 1.17 0.57 0.57 
Sum of least squares 31.44 14.56 21.56 


Indeed, in both cases (Hellinger and mutual information), U;, U2 (high f- 
information value) can be considered as more similar than U; and U3, whereas the 
least squares value between U,, U2 is the highest, meaning that they differ at most. 

We can observe in Example 7 that the values of f-information differ a lot by 
different choices of the function f. Moreover, it is not easy to interpret the values; 
hence, one is interested in calculating normalized values: 
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Normalized Mutual Information: Studholme [35] proposed a normalized mea- 
sure of mutual information. Normalized f -information is defined by 


Ay(ui) + Ae (ur) 
Tp(ui,u2) 


NI ¢ (uy, U2) 2= 


If uw; = uo, then the normalized f -information is minimal with value 2. 


Entropy Correlation Coefficient: Collignon and Maes [25] suggested the use 
of the entropy correlation coefficient, another form of normalized f-information 
(Table 4): 


21 : 2 
Hy CO(u, 9) = Lt) _—_ = 9 , 
Ay (ui) + Hy (u2) NI ¢ (ui, u2) 
The entropy correlation coefficient is one if wu; = wu and zero if u; and up are 


completely independent. 
Exclusive f-Information: It is defined by 


El ¢ (uy, uz) := Aye (uy) + He (uz) — 20 7 (uy, uz). 


Note that the exclusive f-information is minimal for uv; = up. 


Distance Measures Including Statistical Prior Information 


Most multimodal measures used in literature do not consider the underlying image 
context or other statistical prior information on the image modalities. Recently, 
several groups developed similarity measures that incorporate such information: 


¢ Leventon and Grimson [23] proposed to learn prior information from training 
data (registered multimodal images) by estimating the joint intensity distributions 


Table 4 Comparison of Mutual information (uy, Uz) | (U2, U3) | (U3, U1) 
eeu h Sorapoviest DY Normalized* 2.00 5.32 5.32 
f-information and - - * : : 
f-entropies Entropy correlation coefficient | 1.00 0.38 0.38 
Exclusive* 0.00 2.46 2.46 
Hellinger information (uy, Uz)| (U2, U3)| (U3, U1) 
Normalized* 2.00 4.36 4.36 
Entropy correlation coefficient | 1.00 0.46 0.46 
Exclusive* 0.00 1.34 1.34 


*Normalized and exclusive informations are minimal if the 
images are equal, whereas the entropy correlation coefficient 
is maximal 
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of the training images. Based on this paper, Chung et al. [4] proposed to use 
the Kullback-Leibler distance to compare the learned joint intensity distribution, 
with the joint intensity distribution of the images, in order to compare multimodal 
images. This idea was extended by Guetter et al. [14], who combine mutual 
information with the incorporation of learned prior knowledge with a Kullback- 
Leibler term. 

As a follow-up of their ideas, we suggest the following type of generalized 
similarity measures: Let pi. be the learned joint intensity density (learned from 
the training data set) and a € [0, 1]. For f € Fo, define 


Sao (ui, Ur) = aD (Ph, Po (ur, ur)) + (1 — a) Dp (po (ur, 2), Po (ur) Po (u2)) - 
ee 
Tf (uy U2) 
¢ Instead of using a universal but a priori fixed similarity measure, one can learn 
a similarity measure in a discriminative manner. The methodology proposed by 


Lee et al. [22] uses a learning algorithm that constructs a similarity measure 
based on a set of preregistered images. 


3 Mathematical Models for Variational Imaging 


In the following, we proceed with an abstract setting. We are given a physical model 
F, which in mathematical terms is an operator between spaces U and V. For given 
data v € V, we aim for solving the operator equation 


F(®) =v. 


In general, the solution is not unique, and we aim for finding the solution with 
minimal energy, that is, we aim for a minimizer of the constraint optimization 
problem 

R(®) — min subject to F(®) = v. 


In practice, a complication of this problem is that only approximate (noisy) data 
v> € V of v is available. To take into account uncertainty of the data, it is then 
intuitive to consider the following constrained optimization problem instead: 


R(®) — min subject to | F(®)-v° |’ <6, (5) 


where § is an upper bound for the approximation error v—v’. It is known that solving 
(5) is equivalent to minimizing the Tikhonov functional, 


o> ; | F(®) —v'|* + eR), (6) 


where a > 0 is chosen according to Morozov’s discrepancy principle [19]. 


Distance Measures and Applications to Multimodal Variational Imaging 147 


For the formulation of the constrained optimization problem, the Tikhonov 
method, respectively, it is essential that F(®) and v, v°, respectively, represent 
data of the same modality. If F(®) and the data, which we denote now by w, are 
of different kind, then it is intuitive to use a multimodal similarity measure S’, 
instead of the least squares distance, which allows for comparison of F(@®) and w. 
Consequently, we consider the multimodal variational method, which consists in 
minimization of 


® > Ty ,3(®) := S’(F(®), w’) + aR(®), w> 0. 


In the limiting case, that is, for 5 > 0, one aims for recovering an RS’ -minimizing 
solution @* if 


R(®*) = min{R(®) :  € A} where A = {®: ® = arg min{S"(F(-), w)}}. 


To take into account priors in the Tikhonov regularization, the standard way is 
again by a least squares approach. In this case, for regularization the least squares 
functional 


1 
® > Ri) = 5 || — Boll” 


is added to 5 | F(®)—v° |’ (see, e.g., [8]). In analogy, we consider the regulariza- 
tion functional (6) and incorporate priors by adding generalization of the functional 
R\(@). Taking into account prior information W, which might come, for instance, 
from another modality, this leads to the following class of generalized Tikhonov 
functionals: 


Taig!) := S'(F(®),w’) + aR(®) + BS? (®, Vp). 


Here S” is an appropriate multimodal similarity measure. In the limiting case, that 
is, for § > 0, one aims for recovering an y — RS” S?-minimizing solution @" if 


R(®*) + yS? (Ot, Wo) = min{R(®) + yS?(@*, Wo) : ® € A} where 
A={®:@= argmin {S’(F(-), w)}}. 


The y-parameter balances between the amount of prior information and regulariza- 
tion and satisfies y = limy p+0 a For theoretical results on existence of minimizing 
elements of the functionals and convergence, we refer to [11,30]. 
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(x, y) (x, y) 


Fig. 7 Left: images up, ur, right: deformation field ® 


4 Registration 


In this section, we review variational methods for image registration. This problem 
consists in determining a spatial transformation (vector field) ® that aligns pixels of 
two images up and uy in an optimal way (Fig. 7). We use the terminology reference 
image for up and template image for ur, where both are assumed to be compactly 
supported functions in (2. That is, we consider the problem of determining the 
optimal transformation, which minimizes the functional 


u— S" (ur 0 (id + ®), up). (7) 


To establish the context to the inverse problem setting, we use the setting F(@) = 
ur (id + ®) and w' = up. In general, the problem of minimizing (7) is ill 
posed. The Tikhonov-type variational regularization for registration then consists 
in minimization of the functional 


® > S’ (ur 0 (id + ©), up) + AR() (8) 


(we do not consider constrained registration here, but concentrate on the Tikhonov 
regularization). 

Image registration (also of voxel (3D) data) is widely used in medical imaging, 
for instance, for monitoring and evaluating tumor growth, disease development, and 
therapy evaluation. 

Variational methods for registration differ by the choice of the regularization 
functional 7 and the similarity measure S’. There exists a variety of similarity 
measures that are used in practice. For some surveys, we refer to [17, 27,32]. 

The regularization functional R typically involves differential operators. In 
particular, for nonrigid registration, energy functionals from elasticity theory and 
fluid dynamics are used for regularization. 
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The optimality condition for a minimizer of (8) reads as follows: 
aDo(R(®), V) + Do(S’ (ur o (id + ®), ur), ¥) =0 forall We U, (9) 


where Do(7, WV) denotes the directional derivative of a functional 7 in direction 
W. The left-hand side of the equation is the steepest descent functional of the energy 
functional (8). In the following, we highlight some steepest descent functionals 
according to variational registration methods. 


Example 8 (Elastic Registration with L?-Norm-Based Distance Function). Set 
a=1, S’(y,v2) = 5 \|vi — v||F2- We consider an elastic regularization functional 
of the form 


2 2 Ad .,0.,., pwfod_, Olay 
— u J u J 
R(®) fod Ga = +8(a9 7 ae ) i 


where i, t > 0 are Lamé parameters and = (@!, 7). is adjusted to control 
the rate of growth or shrinkage of local regions within the deforming template, and 
| is adjusted to control shearing between adjacent regions of the template [3]. In 
this case, the optimality condition for minimizing ak(®) + S’(ur o (id + ®)), 
given by (9), is satisfied if ® solves the following PDE: 


HAP(x) + (U + ANV(V - ®(x)) = —-(ur(x + (x)) — ur(x))Vur(x + Ox). 


Here A® = (A®!, A@”) and Da(S" (ur o (id + ®),ur),V) = fo aS" W, 
This partial differential equation is known as linear elastic equation and is derived 
assuming small angles of rotation and small linear deformations. When large 
displacements are inherent, it is not applicable [2, 13, 18,24, 29, 40]. 


Example 9 (Elastic Registration with f -Information). Assume that ks € C'(R, R) 
is some kernel density function. Moreover, let Ko(s, t) = ko(s)k,(t). We pose 
the similarity measure as the f-information between the template and the reference 
image: 


S’ (ur 0 (id + ®), up) = Hy (ur) — If (ur 0 (id + ®), up), 


and set a and 7 as in the previous example. In order to write the derivative of J in 
a compact way, we use the abbreviations ® := id+ ®. The derivative of po (ur o ®) 


with respect to ®, in direction WY, is given by 


Da(Polur 0° &), ¥)(t) = [xe — ur (®(x)))Vur(@(x)) U(x) de 
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and 
Dj(Bo(ur 0 ®, ur), V)(S, 1) 


= [ee — ur (®(x))) ko (t — ur(x))(Vur(®(x))- ¥(x)) dx. 
We use the following abbreviations: 


gi(s i= Polur fe) ©)(5) Po(ur)@) 2(s,t) = Po(ur)(t) 


Po (ur o ©, ur)(S, t) (Bo (ur 0 ®, up)(s,t))?’ 


and 


g3(s,t) -= Dg(po(ur 0 ©), V)(s) pour o ®, ur)(s,t)+ 
Po(ur © ®)(s) Dg (Po (ur 0 ®, ur), V)(s, ft). 


With this, we can calculate the derivative of the f-information to be 
D3Uyz (ur © ®, ur), V) 
= ff Pswotur 0 &).¥) (6) polua)(o.F(e165.0) 
RJR 


+ po(ur © &)(s) po(ur)(t) f'(gi(s, t))g2(s, 1)g3(s,t) dt ds. 


For mutual information, this simplifies to 


1 
Dj(MI(ur 0 ®, ug), V) = [[@ (o ( (ur 0 ®, ur) »¥) oomn() 


Dg(Polur o ®, up), V)(s,t 

a (Pa(uro®.ur),Y(S0)\ 1. 1 
Polur © ®)(S) Po(ur)(t) 

A detailed exposition on elastic registration with mutual information can be found 

in [9, 11, 16]. 


In this section, we have presented a general framework on variational-based 
techniques for nonconstrained multimodal image registration. Below we give a short 
overview on relevant literature on this topic. 

Kim and Fessler [21] describe an intensity based image registration technique 
that uses a robust correlation coefficient as a similarity measure for images. It is 
less sensitive to outliers that are present in one image, but not in the other. Kaneko 
[20] proposed the selective correlation coefficient, as an extension of the correlation 
coefficient. Van Elsen et al. investigated similarity measures for MR and CT images. 
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She proposed to calculate the correlation coefficient of geometrical features [37]. 
Alternatively to the correlation coefficient, one could calculate Spearman’s rank 
correlation coefficient (also known as Spearman’s p), which is a nonparametric 
measure of correlation [10], but not very popular in multimodal imaging. Roche 
et al. [33,34] tested the correlation ratio to align MR, CT, and PET images. Woods 
et al. [42] developed an algorithm based on this measure for automated aligning and 
re-slicing PET images. Independently, several groups realized that the problem of 
registering two different image modalities can be cast in an information theoretic 
framework. Collignon et al. [25] and Studholme et al. [35] both suggested using the 
joint entropy of the combined images as a registration potential. Pluim et al. [28] 
investigated in more general f-informations. For MR-CT registrations, the learned 
similarity measure by Lee et al. outperforms all standard measures. Experimental 
results for learning similarity measures for multimodal images can be found in [22]. 


5 Recommended Reading 


For recent results on divergences and information measures, we refer to Computa- 
tional Information Geometry. Website: http://informationgeometry.wordpress.com/ 
(last accessed 5 June 2014). 

Comparison and evaluation of different similarity measures for CT, MR, and PET 
brain images can be found in [41]. 

It is worth mentioning the Retrospective Image Registration Evaluation Project. 
It is designed to compare different multimodal registration techniques. It involves 
the use of a database of image volumes, commonly known as the “Vanderbilt 
Database,” on which the registrations are to be performed. Moreover, it provides 
a training data set for multimodal image registration. Link: http://www.insight- 
journal.org/rire/ (last accessed 5 June 2014). For a collection of databases, we refer 
to the Validation of Medical Image Registration page http://www.vmip.org (last 
accessed 5 June 2014). 

A number of image registration software tools have been developed in the last 
decade. The following support multimodal image comparison: 


¢ ITK is an open-source, cross-platform system that provides developers with an 
extensive suite of software tools for image analysis. It supports the following 
similarity measures: mean squares metric, normalized cross-correlation metric, 
mean reciprocal square differences, mutual information (different implementa- 
tions [26, 39]), the Kullback-Leibler distance, normalized mutual information, 
correlation coefficient, kappa statistics (for binary images), and gradient differ- 
ence metric. Website: www.itk.org/ (last accessed 2 June 2014). 

e FLIRT is a robust and accurate automated linear (affine) registration tool based 
around a multi-start, multi-resolution global optimization method. It can be used 
for inter- and intra-modal registration with 2D or 3D images. Websites: http://fsl. 
fmrib.ox.ac.uk/fsl/fslwiki/FLIRT (last accessed 2 June 2014). 
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e FAIR, FLIRT are toolboxes for fast and flexible image registration. Both 
have been developed by the SAFIR-research group in Liibeck. They include 
the sum of squared differences, mutual information, and normalized gradi- 
ent fields. Websites: http://www.mic.uni-luebeck.de/de/people/jan-modersitzki/ 
software.html (last accessed 4 June 2014). 

e AIR stands for automated image registration. It supports standard deviation of 
ratio images, least squares, and least squares with global intensity rescaling. 
Website: http://bishopw.loni.ucla.edu/AIR5/ (last accessed 5 June 2014). 

¢ RView: This software integrates a number of 3D/4D data display and fusion 
routines together with three-dimensional rigid volume registration using normal- 
ized mutual information. It also contains many interactive volume segmentation 
and painting functions for structural data analysis. Website: http://www.colin- 
studholme.net/software/software.html (last accessed 5 June 2014). 


6 Conclusion 


A number of different similarity measures which are used for comparison of 
multimodal images are listed in this survey. These measures can be grouped 
into four different categories depending on the way the images are interpreted: 
If images are represented by pixel intensities (for instance, representing photon 
counts), one can apply standard /?, L? — norms for comparison. However, these 
distance measures are not recommendable for multimodal images. Viewing images 
as random variables, distance measures from statistics can be utilized. Other 
common measures are based on image morphology (level lines or gradient fields), 
which have the advantage that they are less sensitive (or even blind) to gray-level 
transformations. In applications, images of different modalities neither reveal the 
same morphology nor the positions of edges are matching. Thus, none of the 
above mentioned distance measures are useful. For these applications, comparison 
measures for the gray value distributions of the images were proposed, and in 
order to obtain estimates of these distributions, kernel density estimations have been 
implemented on top. Then, one can either compare the distributions of the images 
directly or alternatively quantify the information content of the joint measures 
in comparison with the product measures. Such distances are called information 
measures. Recent approaches are concerned with learning similarity measures with 
statistical prior information. Variational image registration is discussed, where the 
appropriate distance measure is important in recovering the right local transforma- 
tion parameters. 
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Abstract 

Energy minimization methods are a very popular tool in image and signal 
processing. This chapter deals with images defined on a discrete finite set. 
The energies under consideration can be differentiable or not or convex or not. 
Analytical results on the minimizers of different energies are provided that reveal 
salient features of the images recovered in this way, as a function of the shape of 
the energy itself. An intrinsic mutual relationship between energy minimization 
and modeling via the choice of the energy is thus established. Examples and 
illustrations corroborate the presented results. Applications that take benefit from 
these results are presented as well. 


1 Introduction 


In numerous applications, an unknown image or a signal u, € R? is represented by 
data v € R? according to an observation model, called also forward model 


v = A(u,) with noise, (1) 


where A : R? > R? isa (linear or nonlinear) transform. When u is an m xn image, 
its pixels are arranged columnwise into a p-length real vector, where p = mn and 
the original u[i, 7] is identified with u[(i — 1)m + j]. Some typical applications are, 
for instance, denoising, deblurring, segmentation, zooming and super-resolution, 
reconstruction in inverse problems, coding and compression, feature selection, and 
compressive sensing. In all these cases, recovering a good estimate iu for uy needs 
to combine the observation along with a prior and desiderata on the unknown u,. A 
common way to define such an estimate is 


Find a suchthat (u,v) = minF (w, v), (2) 
ue 
Flu, v) = V(u, v) + BO), (3) 


where F : R? x R? — R is called an energy (or an objective), U C R? is a set of 
constraints, W is a data-fidelity term, ® brings prior information on u,, and 6 > 0 
is a parameter which controls the trade-off between WV and ®. 

The term W ensures that i satisfies (1) quite faithfully according to an appropriate 
measure. The noise n is random and a natural way to derive VW from (1) is to use 
probabilities; see, e.g., [7,32, 37,56]. More precisely, if z(v|u) is the likelihood of 
data v, the usual choice is 


W(u, v) = —logxz(v|u). (4) 


For instance, if A is a linear operator and vy = Au-+-n where n is additive independent 
and identically distributed (i.i.d.) zero-mean Gaussian noise, one finds that 
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p) 
(u,v) x || Au — v|l3. (5) 


This remains quite a common choice partly because it simplifies calculations. 

The role of ® in (3) is to push the solution to exhibit some a priori known or 
desired features. It is called prior or regularization or penalty term. In many image 
processing applications, ® is of the form 


r 


O(u) = ) > H([Diull). (6) 


i=l 


where for any? € {1,...,r}, D; : R? — R’, for s an integer s > 1, are linear 
operators and || - || is usually the 2; or the £2 norm. For instance, the family {D;} = 
{D; :€ {1,...,7}} can represent the discrete approximation of the gradient or the 
Laplacian operator on u or the finite differences of various orders or the combination 
of any of these with the synthesis operator of a frame transform or the vectors of the 
canonical basis of R”. Note that s = 1 if {D;} are finite differences or a discrete 
Laplacian; then 


s=1 => ¢([Djull) = ¢(Dju)). 


And if {D;} are the basis vectors of R”, one has $(|D;u|) = ¢(|u[i]|). In (6), ¢ : 
R; + R is quite a “general” function, often called a potential function (PF). A 
very standard assumption is that 


H1 ¢: Ry — R is proper, lower semicontinuous (l.s.c.) and increasing on Rx, 


with @(t) > o(0) for anyt > 0. 


Some typical examples for ¢ are given in Table | and their plots in Fig. 1. 


Remark 1. If ¢/(0*) > 0 the function t — ¢(|t|) is nonsmooth at zero in which 
case ® is nonsmooth on U;_,[w € R? : Djw = 0]. Conversely, ¢’(0*) = 0 leads 
to a smooth at zero t > ¢(|t|). With the PF (f13), ® leads to the counting function, 
commonly called the £9-norm. 


For the human vision, an important requirement is that the prior ® promotes 
smoothing inside homogeneous regions but preserves sharp edges. According to a 
fine analysis conducted in the 1990s and summarized in [7], ¢ preserves edges if H1 
holds as if H2, stated below, holds true as well: 


H2 lim 


too 


eat 
t 


This assumption is satisfied by all PFs in Table | except for (f1) in case if a = 2. 
Note that there are numerous other heuristics for edge preservation. 
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Table 1 Commonly used PFs ¢ : Ri — R where a > 0 is a parameter. Note that among the 
nonconvex PFs, (£8), (£10), and (f12) are coercive, while the remaining PFs, namely, (f6), (£7), (£9), 
(f11), and (£13), are bounded. And all nonconvex PFs with ¢’ (OT) > Oare concave on R+. Recall 
that (f6) is the discrete equivalent of the Mumford-Shah (MS) prior [17,72] 

Convex PFs 

y’ OT) =0 gy’ (0T) > 0 

(fl) (tt) =t%, l<a<2 (f5) d(t) =¢ 

(f2) d(t) = Ja+?? 

(£3) $(t) = log(cosh(at)) 

(f4) @(t) = t/a — log (1 + t/a) 


Nonconvex PFs 


¢’(0T) =0 ¢’(0T) >0 
(£6) b(t) = minfat?, 1} (f10) p(t) = t%, O0<a<1 
6) = (F11) g(t) = — 
1+ at2 l+at 
(£8) 6(t) = log(at? + 1) (£12) $(t) = log (at + 1) 
(£9) @(t) = 1 — exp (—at?) (£13) $(0) = 0, $(t) = 1ift £0 


Convex PFs Nonconvex PF's 


Fig. 1 Plots of the PFs given in Table 1. PFs with #’(0*) = 0 (- - -), PFs with #’(0*) > 0(—) 


Background 


Energy minimization methods, as described here, are at the crossroad of several 
well-established methodologies that are briefly sketched below. 


e Bayesian maximum a posteriori (MAP) estimation using Markov random field 
(MRF) priors. Such an estimation is based on the maximization of the posterior 
distribution z(u|v) = 2(v|u)2(u)/Z, where z(u) is the prior model for u, and 
Z = m(v) can be seen as a constant. Equivalently, i minimizes with respect to u 
the energy 


F(u,v) = —Inz(v|u) — Inz(u). 
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Identifying these first term above with Y(-, v) and the second one with ® shows 
the basis of the equivalence. Classical papers on MAP energies using MRF priors 
are [14-16, 20, 51,56]. Since the pioneering work of Geman and Geman [56], 
various nonconvex PFs ¢ were explored in order to produce images involving 
neat edges; see, e.g., [54, 55,65]. MAP energies involving MRF priors are also 
considered in many books, such as [32, 53, 64]. For a pedagogical account, 
see [96]. 

¢ Regularization for ill-posed inverse problems was initiated in the book of 
Tikhonov and Arsenin [93] in 1977. The main idea can be stated in terms of 
the stabilization of this kind of problems. Useful textbooks in this direction are, 
e.g., [61, 69, 94] and especially the recent [91]. This methodology and its most 
recent achievements are nicely discussed from quite a general point of view in 
Chapter > Regularization Methods for Ill-Posed Problems in this handbook. 

e Variational methods are related to PDE restoration methods and are naturally 
developed for signals and images defined on a continuous subset Q C R¢?,d = 
1,2,...; for images d = 2. Originally, the data-fidelity term is of the form (5) 
for A = Id and ®(u) = [, (|| Dull2)dx, where @ is a convex function as those 
given in Table | (top). Since the beginning of the 1990s, a remarkable effort 
was done to find heuristics on ¢@ that enable to recover edges and breakpoints 
in restored images and signals while smoothing the regions between them; see, 
e.g., [7, 13, 26, 31, 59, 64, 73, 85, 87]. One of the most successful is the Total 
Variation (TV) regularization corresponding to @(t) = ¢t, which was proposed 
by Rudin, Osher, and Fatemi in [87]. Variational methods were rapidly applied 
along with data-fidelity terms YW. The use of differential operators D* of various 
orders k > 2 in the prior ® has been recently investigated; see, e.g., [22, 23]. 
More details on variational methods for image processing can be found in several 
textbooks like [3, 7,91]. 

For numerical implementation, the variational functional is discretized and ® 
takes the form of (6). Different discretization approaches are considered; see, 
e.g., (2,27, 95] 


The equivalence between these approaches has been considered in several 
seminal papers; see, e.g., [37,63]. The state of the art and the relationship among all 
these methodologies are nicely outlined in the recent book of Scherzer et al. [91]. 
This book gives a brief historical overview of these methodologies and attaches a 
great importance to the functional analysis of the presented results. 


The Main Features of the Minimizers as a Function of the Energy 


Pushing curiosity ahead leads to various additional questions. One observes that 
frequently data fidelity and priors are modeled separately. It is hence necessary to 
check if the minimizer a of F(-,v) obeys all information contained in the data 
model Y as well as in the prior ®. Hence the question: how the prior ® and the 
data-fidelity W are effectively involved in & — a minimizer of F(-,v). This leads to 
formulate the following inverse modeling problem: 
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Analyze the mutual relationship between the salient features exhibited by 
the minimizers # of an energy F(-,v) and the shape of the energy itself. 


(7) 


This problem was posed in a systematic way and studied since [74,75]. The point 
of view provided by (7) is actually adopted by many authors. Problem (7) is totally 
general and involves crucial stakes: 


¢ It yields rigorous and strong results on the minimizers a. 

¢ Such a knowledge enables a real control on the solution — the reconstructed image 
or signal iz. 

¢ Conversely, it opens new perspectives for modeling. 

e It enables the conception of specialized energies F that fulfill the requirements 
in applications. 

e This kind of results can help to derive numerical schemes using knowledge on 
the solutions. 


Problem (7) remains open. The results presented here concern images, signals, and 
data living on finite grids. In this practical framework, the results in this chapter are 
quite general since they hold for energies F which can be convex or nonconvex or 
smooth or nonsmooth, and results address local and global minimizers. 


Organization of the Chapter 


Some preliminary notions and results that help the reading of the chapter are 
sketched in Sect. 2. Section 3 is devoted to the regularity of the (local) minimizers 
of F(-,v) with a special focus on nonconvex regularization. Section 4 shows 
how edges are enhanced using nonconvex regularization. In Sect.5 it is shown 
that nonsmooth regularization leads typically to minimizers that are sparse in the 
space spanned by {D; }. Conversely, Sect. 6 exhibits that the minimizers relevant to 
nonsmooth data fidelity achieve an exact fit for numerous data samples. Section 7 
considers results when both W and ® are nonsmooth. Illustrations and applications 
are presented. 


2 Preliminaries 


In this section we set the notations and recall some classical definitions and results 
on minimization problems. 


Notation 


We systematically denote by a a (local) minimizer of F(-,v). It is explicitly 
specified when u is a global minimizer. 
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* Dj! —The differential operator of order n with respect to the jth component of a 
function. 

¢ yi] — The ith entry of vector v. 

e¢ #J —The cardinality of the set J. 

« J° =I\J —Thecomplement of J Cc J in J where J is a set. 

¢ K+—The orthogonal complement of a sub-vector space K C R". 

¢ A*— The transpose of a matrix (or a vector) where A is real valued. 

« A>0O(A = 0)-The matrix A is positive definite (positive semi-definite) 

¢ 1, € R" —The n-length vector composed of ones, ie., 1,[i] = 1,1 <i <n. 


¢ LL" — The Lebesgue measure on R”. 

¢ Id— The identity operator. 

*  ||.I|) — A vector or a matrix p-norm. 

Rp @ fe R:t>zO}andR* “eR: 1>0}. 


¢ TV — Total Variation. 
* {e1,...,@,} — The canonical basis of R”, ie., e;[7] = 1 and e;[j] = Oifi A 7. 


Reminders and Definitions 


Definition 1. A function F : R? > Riscoerciveif lim F(u) = +00. 


||u|| oo 


A special attention being dedicated to nonsmooth functions, we recall some basic 
facts. 


Definition 2. Given v € RY%, the function F(-,v) : R? — R admits ata € R? a 
one-sided derivative in a direction w € R?, denoted 6, F (i, v)(w), if the following 
limit exists: 


F(u+tw,v) — F(u, v) 
; ; 


iF (a, v)(w) = lim 


where the index | in 6; means that derivatives with respect to the first variable of F 
are addressed. 


Here 6,7 (i,v)(w) is a right-side derivative; the left-side derivative is 
—6,F (iu, v)(—w). If F(-, v) is differentiable at a, then 6,7 (a, v)(w) = D,F (i, v)w 
where D, stands for differential with respect to the first variable (see paragraph 
“Notation”). For ¢ : Ry — R, we denote by ¢'(t~) and ¢’(t*) its left-side and 
right-side derivatives, respectively. 

The classical necessary condition for a local minimum of a (nonsmooth) function 
is recalled [60, 86]: 


Theorem 1. [f F(-,v) has a local minimum at ti € R?, then 6,F (a, v)(w) = 0, for 
every w € R?. 
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If F(-,v) is Fréchet differentiable at #, one finds D; F(a, v) = 0. 

Rademacher’s theorem states that if F is proper and Lipschitz continuous on R?, 
then the set of points in R? at which F is not Fréchet differentiable form a set of 
Lebesgue measure zero [60, 86]. Hence F(-,v) is differentiable at almost every u. 
However, when F(-, v) is nondifferentiable, its minimizers are typically located at 
points where F(-, v) is nondifferentiable; see, e.g., Example | below. 


1 
Example I. Consider F (u,v) = lle —v||? + Blu| for 8 > 0 and u,v € R. The 


minimizer w of F(-,v) reads as 


0 if |v| < B 


i is shrunk w.r.t. v. 
j= sentog it lS é (a is shrunk w.r.t. v.) 


i= 
Clearly, F(-,v) is not Fréchet differentiable only at zero. For any |v| < 6, the 
minimizer of F(-, v) is located precisely at zero. 


The next corollary shows what can happen if the necessary condition in 
Theorem | fails. 


Corollary 1. Let F be differentiable on (IR? x R‘) \ Qo where 
Oo = {(u,v) ER? x RY: Awe R?, —8,F(u,v)(—w) > 8\F(u,v)(w)}. (8) 
Given v € R4, if i is a (local) minimizer of F(-,v) then 
(i, v) ¢ Oo. 
Proof. If wis a local minimizer, then by Theorem 1, 5; (u, v)(—w) = 0, hence 
— 6, F(a, v)(—w) <0 < 6; F(a, v)(w), Vwe R?. (9) 
If (u, v) € Qo, the necessary condition (9) cannot hold. Oo 


Example 2. Suppose that W in (3) is a differentiable function for any v € R?. Fora 


finite set of positive numbers, say 6),..., ,, suppose that the PF ¢ is differentiable 
on Ry \ Uk _ 10; and that 
g(6;)>9' (OF), 1<7<k (10) 


Given a (local) minimizer #, denote 


T=({1,...,r} and I; = {i € 1: ||Djap =6;,,1< 7 <k}. 
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Define F(u,v) = U(u,v) + B x» (||D;u||2), which is differentiable at i. 
iel\ly 

Clearly, F(a#,v) = F(u,v) + B = (||D;%||2). Applying the necessary condition 
ie] 


(9) for w = i yields 


BY ¢ (\IDiallz) <-Di FG, v)(@) < BY ¢’ (IDiall7). 


i€l; iel, 


In particular, one has 7; ¢;, ¢’ (||Dit#llz) < Vier, (|D; #7), which contradicts 
the assumption on ¢’ in (10). It follows that if a is a (local) minimizer of F(-, v), 
then J; = @ and 


|Djall2#0;, 1<j<k, Viel. 


A typical case is the PF (f6) in Table 1, namely, 6(t) = min{at?, 1}. Then k = 1 
and 6; = Te 


The following existence theorem can be found, e.g., in the textbook [35]. 


Theorem 2. For v € RY‘, let U C R? be a nonempty and closed subset and 
F(-,v):U — Ra lower semicontinuous (1.s.c.) proper function. If U is unbounded 
(with possibly U = R?), suppose that F(-,v) is coercive. Then there exists i € U 
such that F(a, v) = inf Fu, v). 


This theorem gives only sufficient conditions for the existence of a minimizer. 
They are not necessary, as seen in the example below. 


Example 3. Let F : R? x R? > R involve (f6) in Table | and read 
F u,v) = (uf1]—v[1])? +86 (| u[1]—u[2]|) for o(t) = maxfat?, 1}, 0 < B <oo. 


For any v, F(- , v) is not coercive since it is bounded by in the direction spanned by 
{(0, u[2])}. However, its global minimum is strict and is reached for #[1] = a[2] = 
v[1] with F(a, v) = 0. 


To prove the existence of optimal solutions for more general energies, we refer 
to the textbook [9]. 

Most of the results summarized in this chapter exhibit the behavior of the 
minimizer points # of F(-,v) under variations of v. In words, they deal with local 
minimizer functions. 


Definition 3. Let F : R? x R4 > Rand O C RY‘. We say that : O > R? isa 
local minimizer function for the family of functions F(-,O) = {F(-,v):v€ O} 
if for any v € O, the function F(-,v) reaches a strict local minimum at U(v). 
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When F(-,v) is proper, l.s.c., and convex, the standard results below can be 
evoked; see [35,49]. 


Theorem 3. Let F(-,v) :R? — R be proper, convex, l.s.c., and coercive for every 
ve R’. 


(i) Then F(-,v) has a unique (global) minimum which is reached for a closed 
convex set of minimizers {a @} eR? :F (u,v) = inf Flu, ol 
ue 


(ii) If in addition F(-,v) is strictly convex, then there is a unique minimizer ti = 
U(v) (which is also global). So F(R’, v) has a unique minimizer function v > 
U(v). 


The next lemma, which can be found, e.g., in [52], addresses the regularity of 
the local minimizer functions when ¥ is smooth. It can be seen as a variant of the 
implicit functions theorem. 


Lemma 1. Let F be C”, m = 2, on a neighborhood of (it, v) € R? x R14. Suppose 
that F(-,v) reaches at ta local minimum such that DIF v) > 0. Then there are 
a neighborhood O C R‘ containing v and a unique C"“! local minimizer function 


U:O—>R?, such that D?F(U(v), v) > 0 for every v € O andU(v) = in. 
This lemma is extended in several directions in this chapter. 


Definition 4. Let ¢ : [0, +co) — R and m = 0 an integer. We say that ¢ is C” on 
R +, or equivalently that ¢ € C’”’(R+), if and only if th (|r|) isC” on R. 


By this definition, #’(0) = 0. In Table 1, left, @ € C!(R4) for (fl) if a < 2, 
¢ € C*(R+) for (£4), while for (£2), (£3), and (£7)-(£9) we find @ € C®(R+). 


3 Regularity Results 


Here, we focus on the regularity of the minimizers of F : R? x R? — R of the form 


F(u,v) = |Au—vll3 + BY) d(\|Diull). (11) 
16. 


P= 


where A € R?*? and for anyi € J we have D; € R**? for s > 1. Let us denote by 
D the following rs x p matrix: 
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When A in (11) is not injective, a standard assumption in order to have regulariza- 
tion is 
H3 ker(A) MN ker(D) = {0}. 


H3 is trivial if rank A = p or rankD = p. Often, ker(D) = span(1,) and 
Al, # 0, so H3 holds. 


Some General Results 


We first verify the conditions on F(-, v) in (11) that enable Theorems 2 and 3 to be 
applied. Since H1 holds, F(-,v) in (11) is L.s.c. and proper. 


1. F(-,v) in (11) is coercive for any v € R¢ at least in one of the following cases: 


¢ Rank(A) = p and@¢@: R+ + R4 is nondecreasing. 
¢ HI and H3 hold and lim; 7. d(t) = o (e.g., (f1)—-(f5),(£8), (f10), and (£12) 
in Table 1). 


By Theorem 2, F(-,v) has minimizers. 

2. For any v € R‘, the energy F(-,v) in (11) is convex and coercive if H1 and H3 
hold for a convex @. Then the claim in Theorem 3(3) holds true. 

3. Further, F(-,v) in (11) is strictly convex and coercive for any v € R¢ if ¢ 
satisfies H1 and if one of the following assumptions holds: 


¢ Rank(A) = p and @¢ is convex. 
¢ H3 holds and ¢ is strictly convex. 


Then the claim in Theorem 3(3) holds. Further, if F is C” for m > 2, then the 
minimizer function U/ : RY + R? (see Definition 3) is C’”—'! by Lemma 1. 


However, the PFs involved in (11) used for signal and image processing are often 
nonconvex, bounded, or nondifferentiable. One extension of the standard results is 
given in the next section. 


Stability of the Minimizers of Energies with Possibly Nonconvex 
Priors 


Related questions have been considered in critical point theory, sometimes in semi- 
definite programming; the well-posedness of some classes of smooth optimization 
problems was addressed in [42]. Other results have been established on the stability 
of the local minimizers of general smooth energies [52]. Typically, these results are 
quite abstract to be applied directly to energies of the form (11). 

Here the assumptions stated below are considered. 
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H4 The operator A in (11) satisfies rank A = p, i.e., A* A is invertible. 
H5 The PF ¢ in (11) is C°(R4) andC”, m = 2, on R4 withO < ¢'(0T) < 0. 


Under H1, H2, H4, and HS, the prior ® (and hence F(-,v)) in (11) can be 
nonconvex and in addition nonsmooth. By H1 and H4, F(-,v) in (11) admits a 
global minimum v € R? — see Item | in section “Some General Results.” However, 
F(-,v) can present numerous local minima. 


bm Energies F with nonconvex and possibly nondifferentiable PFs ¢ are fre- 
quently used in engineering problems since they were observed to give rise to 
high-quality solutions i. It is hence important to have good knowledge on the 
stability of the obtained solutions. 


The results summarized in this section provide the state of the art for energies of 
the form (11). 


Local Minimizers 
The stability of local minimizers is an important matter in its own right for several 
reasons. Often, a nonconvex energy is minimized only locally, in the vicinity of 
some initial guess. Second, the minimization schemes that guarantee the finding 
of the global minimum of a nonconvex objective function are exceptional. The 
practically obtained solutions are usually only local minimizers. 

The statements below are a simplified version of the results established in [44]. 


Theorem 4. Let F(-,v) in (11) satisfy HI, H2, H4, and H5. Then there exists a 
closed subset © C R4 whose Lebesgue measure is L4(@) = 0 such that for any 
v € R¢ \ ©, there exists an open subset O C R4 with v € O and a local minimizer 
function (see Definition 3) U : O — R? which is C™~! on O and fulfills i = U(v). 

Since © is closed in R? and L4@) = 0, the stated properties are generic. 
Commentary on the Assumptions 
All assumptions H1, H2, and H5 bearing on the PF ¢ are nonrestrictive; they address 
all PFs in Table | except for (f13) which is discontinuous at zero. The assumption 
H4 cannot be avoided, as seen in Example 4. 
Example 4. Consider F : R* x R > R given by 

F (u,v) = (uf 1] — u[2] — v)? + lull] + lul2]1, 


where v = v[1]. The minimum is obtained after a simple computation. 


1 1 1 
v> 5 “= («. c-—Vvt+ 5) forany c € o. v— 5| (nonstrict minimizer). 
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a = 0 (unique minimizer) 
. 1 1 . ae 
v<-- u={c,c-v— 5 forany c€}v+ rt O| (nonstrict minimizer). 


In this case, assumption H4 fails and there is a local minimizer function only for 


Other Results 
The derivations in [44] reveal several other practical results. 


1. If @ € C?(R4), see Definition 4, then Vv € R? \ ©, every local minimizer @ of 
Fu, v) is strict and D?F (ii, v) > 0. Consequently, Lemma | is extended since 
the statement holds true Vv € R? \ ©. 


> For real data v—a random sample of R¢ — whenever F(-, v) is differentiable 
and satisfies the assumptions of Theorem 4, it is a generic property that 
local minimizers i are strict and their Hessians Di F (itv) are positive 
definite. 


2. Using Corollary 1, the statement of Theorem 4 holds true if ¢’(0*) = 0 and if 
there is t > O such that f’(t~) > ¢’(r*). This is the case of the PF (f6) in 
Table 1. 

3. If ¢’(0T) > 0, define 

SG er : Dw =0} and K; 2 |weR? : Dw = 0, Vi eS. 
(12) 
Then Vv € R¢ \ ©, every local minimizer u of F(u, v) is strict and 
(a) DiF|x,(a@,v) = 0 and Di F\x; (i, v) > 0 —a sufficient condition for a 
strict minimum on K ;. 
(b) 6;5F (a, v)(w) > 0, Vw e K;\{0} — a sufficient condition for a strict 


minimum on K - 


pm Here (a) and (b) provide a sufficient condition for a strict (local) minimum 
of F(-,v) at a (a direct consequence of [80, Theorem 1]). These conditions 
are satisfied at the (local) minimizers a of F(-,v) for every v € R4, except 
for a negligible subset of R?, in which case Lemma 1 can be applied. 


One can interpret these results as follows: 


pm Under the assumptions H1, H2, H4, and H5, given real data v € R?, the 
chance to get a nonstrict (local) minimizer or a (local) minimizer of the 
energy in (11) that does not result from a C’”—! local minimizer function 
is null. 
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Global Minimizers of Energies with for Possibly Nonconvex Priors 
The results on the global minimizers of (11) presented next are extracted from [45]. 


Theorem 5. Assume that F(-,v) in (11) satisfy H1, H2, H4, and H5. Then there 
exists a subset © C RY such that L4 (0) = 0 and the interior of R‘ \ © is dense 
in IR‘, and for any v € R¢ \ © the energy F(-,v) has a unique global minimizer. 
Furthermore, the global minimizer function U : R4 \ © > R? isC”™—! onan open 
subset of R41 \ © which is dense in R4. 


> Otherwise said, in a real-world problem there is no chance of getting data v 
such that the energy F(-,v) (11) has more than one global minimizer. 


Nonetheless, © plays a crucial role for the recovery of edges; this issue is 
developed in Sect. 4. 


Nonasymptotic Bounds on Minimizers 


The aim here is to give nonasymptotic analytical bounds on the local and the global 
minimizers u of F(-,v) in (11) that hold for all PFs ¢ in Table 1. Related questions 
have mainly been considered in particular cases or asymptotically; see, e.g., [4, 71, 
92]. In [51] the mean and the variance of the minimizers a for strictly convex and 
differentiable functions ¢ have been explored. 

The bounds provided below are of practical interest for the initialization and 
the convergence analysis of numerical schemes. The statements given below are 
extracted from [82]. 


Bounds on the restored data. One compares the “restored” data Au with the given 
data v. 


H6 Consider the alternative assumptions: 
* $/(0t) = Oand@ € C!(Ry\o) where the set Oy = {t > 0: b'(t~) > b'(t*)} 


is at most finite. 


* $'(0T) > Oand ¢ is C' onR*. 


The set ©o allows us to address the PF given in (f6). Let us emphasize that under 
H1 and H6, the PF ¢ can be convex or nonconvex. 


Theorem 6. Consider F(-,v) of the form (11) where H1, H3, and H6 hold. For 
every v € R®, if F(-,v) has a (local) minimum at i, then 


|| Adll2 < [Iv le. 


Comments on the results. This bound holds for every (local) minimizer of 
F(-,v). If A is a uniform tight frame (i.e., A* A = Id), one has 
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l|@ll2 < |Ivll2. 


The mean of restored data. In many applications, the noise corrupting the data 
can be supposed to have a mean equal to zero. When A = Id, it is well known that 
mean(i) =mean(v); see, e.g., [7]. However, for a general A one has 


Al,x1, =  mean(u%) = mean(v). (13) 


The requirement Al, « 1, is quite restrictive. In the simple case when $(t) = 17, 
ker(D) = 1,; and A is square and invertible, it is easy to see that this is also a 
sufficient condition. Finally, if A #4 Id, then generally mean (i) # mean (vy). 


The residuals for edge-preserving regularization. A bound on the data-fidelity 
term at a (local) minimizer u of F(-, v) shall be given. The edge-preserving H2 (see 
Sect. 1) is replaced by a stronger edge-preserving assumption: 


def = 
H7  ||$"lloo = max 5 sup |f(¢7)|, sup |$"(t7)|-_ < 00. 
t20 t>0 


Except for (f1) and (f13), all other PFs in Table | satisfy H7. Note that when 
p y 
¢'(0T) > 0 and H7 hold, one usually has ||¢’ ||. = ¢’(07). 


Theorem 7. Let F(-,v) be of the form (11) where rank (A) = q < p, and H1, H3, 
H6, and H7 hold. For every v € R‘, if F(-,v) has a (local) minimum at i, then 


: B oe 
At — vlloo < FII9'llooll(A4*) 'Alloo||D]l1- (14) 


Let us emphasize that the bound in (14) is independent of data v and that it 
is satisfied for any local or global minimizer i of F(-,v). (Recall that for a real 
matrix C with entries C[i, j], one has ||C |]; = max; >); |C[i, /]| and ||Clloo = 
max; )/; |C[i, j]|; see, e-g., [35].) 

If D corresponds to a discrete gradient operator for a two-dimensional image, 
||D||; = 4. If in addition A = Id, (14) yields 


Iv — alloc < 2B119'lloo. 


The result of this theorem may seem surprising. In a statistical setting, the 
quadratic data-fidelity term || Au — v||> in (11) corresponds to white Gaussian noise 
on the data, which is unbounded. However, if @ is edge preserving according to 
H7, any (local) minimizer # of F(-,v) gives rise to a noise estimate (v — Au)i], 
1 <i <q thatis tightly bounded as stated in (14). 


b> Hence the model for Gaussian noise on the data v is distorted by the solution 


it. 
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pm When F(-,v) is convex and coercive, (14) shows that a good initialization for 
a minimization algorithm should be a point up such that Aug = v, e.g., the 
minimum norm solution of ||v — a||2 given by uy = A*(AA*)~!v. 


4 Nonconvex Regularization 
Motivation 


A permanent requirement is that the energy F favors the recovery of neat edges. 
Since the pioneering work of Geman and Geman [56], various nonconvex ® in (3) 
have been proposed [15,54,55,64,68,72,85]. Indeed, the relevant minimizers exhibit 
neat edges between homogeneous regions. However, these nonconvex energies are 
tiresome to control and to minimize (only few algorithms are proved to find the 
global minimizer in particular cases). In order to avoid these numerical intricacies, 
since the 1990s, an important effort was done to derive convex edge-preserving PFs; 
see, e.g., [20,31,57,64, 87] and [7] for an overview. The most popular convex edge- 
preserving PF was derived by Rudin, Osher, and Fatemi [87]: it amounts to ¢ = f, 
for {D;} yielding the discrete gradient operator, the £2-norm in (6) (see Sect. 1), and 
the relevant ® is called the Total Variation (TV) regularization. 

In Fig.2 one sees that the height of the edges is better recovered when ¢ is 
nonconvex, compared to the convex TV regularization. The same effect can also 
be observed, e.g., in Figs. 7, 8, and 10. 

A considerable progress in nonconvex minimization has been realized. For 
energies of the form (2)-(3) we refer to [5, 19, 88, 89]. 


» This section is devoted to explain why edges are nicely recovered using a 
nonconvex ¢. 
Assumptions on Potential Functions ¢ 


Consider F(-,v) of the form (11) where D; :R? > R',i eI = {W557} Les 


F(u,v) = ||Au—v|l3 + B >) 4 (Diul), (15) 


ie] 


and @ : Rt — R4+ satisfies H1 (see Sect. 1), H6 section “Nonasymptotic Bounds 
on Minimizers,” and H8 given below 


H8 ¢ is C* and $'(t) = 0 on R¥,, and inf $"(t) <0 with lim o” (t) = 0; 
teR4 00 


as well as one of the following assumptions: 
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1 100 1 100 


Data v = u +n (—) Convex regularization ¢(t) = t(TV) 
Original u, (dashed line) Original u, (---), minimizer u (—) 


1 100 


Non-convex regularization 


o(t) = at/(1+ at) 


Fig. 2. Minimizers of F(u, v) = |lu — vil} + B SP] @(lufé] — uli + 1) 


H9 ¢/(0*) = 0, and there are two numbers t > 0 and T € (t,00) such that 
ob” (t) = Oon (0, t], 6” (t) < 0 on (t, 00), 6” (t) decreases on (t, T) and increases 
on (T, 00). 


H10 ¢/(0T) > 0, and lim $"(0) <0 is well defined and $"(t) <_ 0 strictly 
t—> 
increases on (0, co). 
These assumptions are illustrated in Fig. 3. They hold for all nonconvex PFs in 
Table 1, except for (f6) and (f13) which are presented separately. Further, these 


assumptions are easy to relax. 
The results presented below come from [81]. 
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¢'(0+) = 0 (@ differentiable) 
(H1, H8 and H9) 
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¢'(0*) > 0 (® non differentiable) 
(H1, H8 and H10) 


increases, <0 


increases, <0 


#0) <0 


10) 1 


oO 1 


Fig. 3 Illustration of the assumptions in two typical cases — (f7) and (f11) — in Table 1 


How It Works on R 


> This example illustrates the main facts that explain why edges are enhanced 
when ¢ is nonconvex, satisfying H1, and H8 along with either H9 or H10. 


Let ¥ : Rx R-— R be given by 


F (u,v) 
. B > sgh 
= = (uv)? + Bou) for 
5 p 


1 
> a nr 
[lim, 0 (2) 


if ¢’(0*) =0 (HI, H8 and H9) 


if $’(0*) > 0 (HI, H8 and H10) 


The (local) minimality conditions for u of F(-,v) read as 


* Ifd/(0T) = Oor[¢’(0*) > Oanda 4 0]: 7+ Bd’ (@) = vand1+ Bp" (i) = 0. 


If d'(0*) > Oanda = 0: |v| < B¢'(0*). 


To simplify, we assume that v = 0. Define 


69 = infCg and 6; = supCg, 


for Cg = {ue R4: DiF(u,v) <0} = {we R46”) < —1/B}. 


One has 6) = 
calculations yield 


0 if #/(0+) > Oand0 < & < T < 6 if #/(0t) = 0. A few 
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. For every v € R+ no minimizer lives in (8, 9) (cf. Fig. 4). 
. One computes 0 < & < & such that (cf. Fig. 4) 
a. If0 <v < &, F(-,v) has a (local) minimizer ai € [0, 40], hence iz is subject 
to a strong smoothing. 
b. If v = &, F(-,v) has a (local) minimizer a, > 6), hence a is subject to a 
weak smoothing. 
c. If v € [&, &] then F(-, v) has two local minimizers, io and i). 
3. There is € € (&, &1) such that F(-,&) has two global minimizers, F (ao, €) = 
F (im, €), as seen in Fig. 5; 
a. If 0 < v < &, the unique global minimizer is u = i. 
b. If v > &, the unique global minimizer is # = 1. 
4. The global minimizer function v +> U/(v) is discontinuous at € and C!-smooth on 


Ry\t&}. 


Ne 


Item | is the key for the recovery of either homogeneous regions or high edges. The 
minimizer io (see Items 2a and 3a) corresponds to the restoration of homogeneous 
regions, while iz; (see Items 2b and 3b) corresponds to edges. Item 3 corresponds to a 
decision for the presence of an edge at the global minimizer. Since {&} is closed and 
L'{€} = 0, Item 4 confirms the results of section “Global Minimizers of Energies 
with for Possibly Nonconvex Priors.” 


Either Smoothing or Edge Enhancement 


(A) Case ¢’(0*+) = 0. Below the case depicted in Figs.4, left, and 5, left, is 
extended to R?. 


F(u,v) for o(t) = # F(u,v) for o(t) = an 


a+t? 


u + B¢'(u) utBP(u) 


¢'(0T) =0 ¢'(0t) >0 
Minimizer if : &+ Bd'(&) =v Minimizer at 0: |v| < B¢/(07) 
and 1+ 6¢"(a) >0 Else : @ + 8¢'(@) = v and 1+ 86"(a) > 0 


Fig. 4 The curve of w+» (D,F(u, v) — v) on R \ {0}. All assumptions mentioned before hold 
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$(u) = Ghawy b(u) = aw 


Fig. 5 Each curve represents F(u, v) = $(u—v)y? + B¢(u) for an increasing sequence v € [0, &). 
The global minimizer of each F(-, v) is marked with “e.” No (local) minimizer lives in (6, 41) 


Theorem 8. Let F(-,v) be of the form (15) where H1, H3, H8, and H9 hold, and 
{D; : i € I} are linearly independent. Set ju = max; <j<; ||D*(DD*)~'e;||2. For 
p> ae there are 0) € (t,T) and 0; € (T, 0c) such that Vv € RY, ifiisa 
(local) minimizer of F(-,v), then 


either |Dju| < %, or |Dju| >, Vi eT. (16) 


In many imaging problems, {D;} are not linearly independents. If {D;} are 
linearly dependent, the result (16) holds true for all (local) minimizers % that are 
locally homogeneous on regions that are connected with respect to {D;}. Otherwise, 
one recovers both high edges and smooth transitions, as seen in Fig. 8a. When @ is 
convex, all edges are smoothed, as one can observe in Fig. 7a. 

The PF ¢(¢) = min{az?, 1} (£6) in Table | does not satisfy assumptions H8 and 
H9. From Corollary | and Example 2 (section “Reminders and Definitions’), any 
(local) minimizer a of F(-,v) obeys 


1 
Du —, Viel. 
Dial # a 


Proposition | below addresses only the global minimizers of F(-,v). 


Proposition 1. Let F(-,v) be given by (15) where $(t) = min{at?, 1}, {D:i € 
I} are linearly independent and rank (A) > p—r = 1. If F(-,v) has a global 
minimizer at t, then 


1 1 
—T;, or |D,u| => ; 
Ja i Ja T; 


either |Dju| < 
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_ || Be; || 
for T; = ,{/—_* 2 <1, viel, (17) 
|| Beil]; + oB 


where B is a matrix depending only on A and D. 


If D = Id, then B = A. If u one-dimensional signal and Dju = u[i] — ufi + 1], 
1 <i < p—1,onehas B = (Id— 5 11")AH where H € R?*? is upper triangular 
composed of ones. 


In Proposition 1, set ) = —— and 6; = : for y = max; < 1 
p' as 0 Ja 1 Jay Y jel l . 
Let us define the following subsets: 


jy 2 ie + |Dja| <%} and FZ I\ fh =fi el : |Djal>O}. (18) 


One can interpret the results of Theorem 8 and Proposition | as follows: 


> The pixels in Jo form homogeneous regions with respect to {D;}, whereas the 
pixels in J, are break points. 
In particular, if {D;} correspond to first-order differences, jy addresses 
smoothly varying regions where |D;%| < 6, while Jy corresponds to edges 
higher than 6; — 4. 


(B) Case ¢’(0*) > 0. Here the results are stronger without assumptions on {D;}. 
This case corresponds to Figs. 4, right, and 5, right. 


Theorem 9. Consider F(-,v) of the form (15) where H3 holds and ¢ satisfies H1, 
2 2 

H8, and H10. Let B > Or LOn where 4 > O is a constant depending only on 

{D;}. Then 46, > 0 such that Vv € R4, every (local) minimizer it of F(-, v) satisfies 


either |Dju|=0, or |Djul|>O, Wiel. (19) 


The results of Theorem 9 were extended to energies involving box constraints in 
[33]. 

The “0-1” PF (f13) in Table 1 does not satisfy H8 and H10 since it is 
discontinuous at 0. 


Proposition 2. Let F(-,v) in (15) be defined for @(0) = 0, o(t) = Lift > 0, i.e, 
(f13), {D;} be linearly independent and rank A = p—r = 1. If F(-,v) has a global 
minimum at ii, then 


vB 


either |Dju| =O or |Dj;a| = , 
|| Bei lo 


Viel, (20) 


where the matrix B depends only on D and on A. 
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In (20), B is the same as in Proposition 1. For 6, = Minje7 wear it is clear that 
(20) holds. 
Let 


Jo 2 fi + [D8] =O} and /, 2 1 \ fo = {i : |D;A| = 41}. 


Using this notation, the results of Theorem 9 and Proposition 2 show that: 


& The indexes in /y) address regions in a that can be called strongly homoge- 
neous (since |D;i| = 0), while J, addresses breakpoints where |D;i| > 01. 

If {D;} are first-order differences, w is neatly segmented: Jo corresponds to 
constant regions, while J, describes all edges and they are higher than 4. 


Direct segmentation of an image from data transformed via a general (nondiag- 
onal) operator A remains a difficult task using standard methods. The result in (19), 
Theorem 9, tells us that such a segmentation is naturally involved in the minimizers 
i of F(-,v), for any operator A. This effect can be observed, e.g., on Figs. 8b, d 
and 11d. 


(C) Illustration: Deblurring of an image from noisy data. The original image 
Uy in Fig. 6a presents smoothly varying regions, constant regions, and sharp edges. 
Data in Fig. 6b correspond to v = a * uy +n, where a is a blur with entries a;,; = 
exp (—(i? + j?)/12.5) for —4 < i,j < 4, and n is white Gaussian noise yielding 
20 dB of SNR. The amplitudes of the original image are in the range of [0, 1.32] 
and those of the data in [—5, 50]. In all restored images, {D;} correspond to the 
first-order differences of each pixel with its 8 nearest neighbors. In all figures, the 
obtained minimizers are displayed on the top. Just below, the sections corresponding 
to rows 54 and 90 of the restored images are compared with the same rows of the 
original image. 

The restorations in Fig.7 are obtained using convex PFs ¢ while those in 
Fig. 8 using nonconvex PFs @. Edges are sharp and high in Fig.8 where @ is 
nonconvex, which corroborates the results in paragraphs (A) and (B). In Fig. 8b, 
d @ is nonconvex and $¢'(0+) > 0 in addition. As stated in Theorem 9, in spite of 
the fact that A is nondiagonal (and ill-conditioned), the restored images are fully 
segmented and the edges between constant pieces are high. 


5 Nonsmooth Regularization 


> Observe that the minimizers corresponding to ¢/(0+) > 0 (nonsmooth 
regularization) in Figs.2b, c, 7b, 8b, d, 10a—c, and 11d are constant on 
numerous regions. This section is aimed to explain and to generalize this 
observation. 
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Fig. 6 Data v = a x uo +n, where a is a blur and n is white Gaussian noise, 20 dB of SNR. (a) 
Original image. (b) Data v= blur + noise 


a (OT) =0 b g(0*) > 0 


Fig. 7 Restoration using convex PFs. (a) ¢(f) = ¢% fora = 1.4,B = 40. (b) d(¢t) = ¢ for 
B = 100 
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¢(0*) > 0 


a ¢(0T) =0 


tow 


se 8 Restoration using nonconvex PFs. (a) @(t) = eer <— 5 fora = 25, B = 35. (b) o(t) = 


— fora = 20, B = 100. (c) o(t) = min{at?, ie fora = 60, B = 10. (d) (0) = 
0, Ot) = 1, t > Ofor B = 25 
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Consider 


F(u,v) = V(u,v) + B®(u) (21) 


P(u) = Do ([Diul,). (22) 


i=1 


where W : R? x R? > Ris any explicit or implicit C"-smooth function for m > 2 
and D; : R? & R*, Vi e J = {1,...,r}, are general linear operators for any 
integer s > 1. It is assumed that ¢ satisfies H1 along with 


H11 ¢ is C*-smooth on R*, and $'(0*) > 0. 


Note that WY and ¢ can be convex or nonconvex. Let us define the set-valued function 
J on R? by 


T(u) = {i E1: |[Dull2 = OF (23) 


Given u € R?’, J(u) indicates all regions where D;ju = 0. Such regions are 
called strongly homogeneous with respect to {D; }. (The adverb “strongly” is used to 
emphasize the difference with just “homogeneous regions” where ||D;u||2 ~ 0.) In 
particular, if {D;} correspond to first-order differences between neighboring samples 
of u or to discrete gradients, 7 (u) indicates all constant regions in u. 


Main Theoretical Result 
The results presented below are extracted from [80]. 


Theorem 10. Given v € R’, assume that F(-,v) in (21)-(22) is such that WV is 
C™, m > 2 on R? x RY, and that @ satisfies HI and H11. Let u € R? be a (local) 


minimizer of F(-,v). For t= J(u), let K; be the vector subspace 
K; = {weR?’ :Du=0,vie J. (24) 


Suppose also that 


(a) 6,F (a, v)(w) > 0, for every w € K; \ {0}. 
(b) There is an open subset 0 C R®¢ such that F |x, e 0’) has a local minimizer 
function U; : 0; — K; which is C™—! continuous and t = U;(v). 


Then there is an open neighborhood O; C 0% of v such that F(-, O;) admits a 
Cc”! local minimizer function U : O; — R? which satisfies U(v) = i, Ulk, =U; 
and 
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veO; > DU(vr) =0, forallie f. (25) 


Note that J and K j are the same as those introduced in (12) section “Local 
Minimizers.” 


Commentary on the assumptions. Since F(-,v) has a local minimum at a, by 
Theorem | one has 6;F(u,v)(w) = 0, for all w € K; \ {0}, and if for some w 
the inequality becomes inequality, then the inequality is strict for —w. So (a) is not 
a strong requirement. Condition (b) amounts to Lemma | (section “Reminders and 
Definitions”) applied to F| x, which is C” on a neighborhood of (u, v) belonging to 
K S x R!. 

If F(-,v) (possibly nonconvex) is of the form (11) and assumption H4 (sec- 
tion “Stability of the Minimizers of Energies with Possibly Nonconvex Priors”) 
holds, Theorem 4 and the other results given next show that (a) and (b) are satisfied 
for any v € R4 \ © where © is closed and L4(©) = 0. 


Significance of the results. Using the definition of 7 in (23), the conclusion of the 
theorem can be reformulated as 


veO; >= TUW)2I & UWE K;. (26) 


Minimizers involving large subsets J are observed in Figs. 2b, c, 7b, 8b, d, 10a—c, 
and 11d. It was seen in Examples 1 and 4, as well as in section “How It Works on 
R” (case ¢’(0+) > 0), that J is nonempty for data v living in an open O j- Note 
also that there is an open subset 0; Cc O; such that 7 (U(v)) = J for all vy € O;. 
These sets O j are described in Example 5. 

Observe that (26) is a severe restriction since K ; is a closed and negligible subset 
of IR’, whereas data v vary on open subsets O; of R?. 

Focus on a (local) minimizer function ’/ : O — R? for F(-, O) and put J= 
J(U(v)) for some v € O. By Theorem 10, the sets O; and 0; are of positive 
measure in IR’. When data v range over O, the set-valued function (7 o//) generally 
takes several distinct values, say {J;}. Thus, with a (local) minimizer function U, 
defined on an open set O, there is associated a family of subsets {0 J, $ which form 
a covering of O. When v € Oy, , we find a minimizer # = U(v) satisfying 7 (a) = 
Jj. 


em Energies with nonsmooth regularization terms, as those considered here, 
exhibit local minimizers which generically satisfy constraints of the form 
T(t) 4G. 

In particular, if {D;} are discrete gradients or first-order difference operators, 
minimizers i are typically constant on many regions. For example, if ¢(r) = f, 
we have ®(u) = TV(u), and this explains the stair-casing effect observed in 
TV methods on discrete images and signals [30, 39]. 
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Examples and Discussion 
The subsection begins with an illustration of Theorem 10 and its meaning. 


Restoration of a noisy signal. Figure 9 shows a piecewise constant signal uo 
corrupted with two different noises. 

Figure 10 depicts the restoration from these two noisy data aca by minimiz- 
ing an energy of the form F(u,v) = ||u — v||? + B aan o(\uli] — ufi + 1]|). The 
minimizers shown in Fig. 10a—c correspond to functions ¢ such that ¢’/(0*) > 0 
and they are constant on large segments. The reader is invited to compare the 
subsets where these minimizers are constant. The function ¢ in Fig. 10d satisfies 
¢'(0*) = 0 and the resultant minimizers are nowhere constant. 

Example 5 below gives a rich geometric interpretation of Theorem 25. 


Example 5 (1D TV Regularization). Let F : R’? x R? —> R be given by 


p-l 
Flu, v) = |/Au—vl5 + B >> luli] ]-—ufi + 1]), 


i=1 


B>0, (27) 


where A € R?*? is invertible. Clearly, there is a unique minimizer function U/ for 
F(-,R?’). Two striking phenomena concerning the sets O,; are described next: 


1. For every point # € R?, there is a polyhedron Q, C R? of dimension #7 (i), 
such that for every v € Q;, the same point U/(v) = @ is the unique minimizer of 
F(-,v). 

2. Forevery J C {l,..., p — 1, there is a subset O; C R?, composed of 2?-#/—! 
unbounded polyhedra (of dimension p) of R? such that for every v € Oy, the 


1 100 1 100 


Fig. 9 Data v = u,-++n (—) corresponding to the original u, (-.-.) contaminated with two different 
noise samples 7 on the /eft and on the right 
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1 100 1 100 


1 100 100 


e : 
ft<a 
th=9? . 1S (here $/(0*)=0 
a(t) oe FE (here $'(0*)=0) 


Fig. 10 Restoration using different functions @. Original u, (-.-.), minimizer u (—). Each figure 
from (a) to (d) shows the two minimizers u corresponding to the two data sets in Fig. 9 (left and 
right), while the shape of ¢ is plotted in the middle 
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minimizer u of F(-,v) satisfies 4; = w+, for alli € J anda; 4 44, for all 
i € J°. Their closure forms a covering of R?. 


The next Remark 2 deserves to be combined with the conclusions of sec- 
tion “Nonasymptotic Bounds on Minimizers.” 


Remark 2. The energy in (27) has a straightforward Bayesian interpretation in 
terms of maximum a posteriori (MAP) estimation (see section “Background,” first 
item). The quadratic data-fidelity term corresponds to an observation model of 
the form v = Au, + n where n is independent identically distributed (i.i.d.) 
Gaussian noise with mean zero and variance denoted by o7. The likelihood reads 


1 
m(v|u) = exp —Fo2 || Au — vi). The regularization term corresponds to an iid. 
o 
Laplacian prior on each difference u[i]—u[i + 1], 1 <i < p—1, that is, exp (—A|t]) 
for A = >~5: Since this density is continuous on R, the probability to get a null 


sample, t = uli |—u[i +1] = 0, is equal to zero. However, the results presented above 
show that for the minimizer a of F(-, v), the probability to have u[i]—a[i + 1] = 0 
for a certain amount of indexes 7 is strictly positive. This means that the Laplacian 
prior on the differences u[i] — u[i + 1] is far from being incorporated in the MAP 
solution i. 


Applications 


The use of nondifferentiable (and also nonconvex) regularization in compressive 
sensing is actually extremely abundant; readers can check, e.g., the textbook [50]. 


Image reconstruction is computed tomography. The concentration of an isotope 
in a part of the body provides an image characterizing metabolic functions and local 
blood flow [21, 62]. In emission computed tomography (ECT), a radioactive drug 
is introduced in a region of the body and the emitted photons are recorded around 
it. Data are formed by the number of photons v[i] = O reaching each detector, 
i = 1,...,q. The observed photon counts v have a Poissonian distribution [21,90]. 
Their mean is determined using projection operators {a;,i = 1,2,...,q} anda 
constant p > 0. The data-fidelity YW derived from the log-likelihood function is 
nonstrictly convex and reads: 


q q 
W(u,v) = of di, ‘ — > vfi] In ((a;.u)). (28) 


i=1 i=1 


Figure 11 presents image reconstruction from simulated ECT data by minimizing 
and energy of the forms (21) and (22) where W is given by (28) and {D;} yield 
the first-order differences between each pixel and its eight nearest neighbors. One 
observes, yet again, that a PF ¢ which is nonconvex with ¢’(0T) > 0 leads to a 
nicely segmented piecewise constant reconstruction. 
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Fig. 11. ECT. F(u,v) = V(u,v) + B Ye, ¢(|Djul). (a) Original phantom. (b) ECT simulated 
data. (c) ¢’(0) = 0, edge preserving. (d) (t) = t/(a + t) (#’(0T) > 0, nonconvex) 


Ler — 


F(u,v) = |lu— vl] + 6|/Dull Fu,v) = |lu—ovlli + Bl|DullZ Fu, v) = |lu— vfl3 + BI|Dull3 
Stair-casing Exact data-fit 


Fig. 12 Disa first-order difference operator, i.e., Dju = u[i]—u[i+1],1 <i < p—1. Data (- --), 
restored signal (—). Constant pieces in (a) are emphasized using “*,” while data samples that are 
equal to the relevant samples of the minimizer in (b) are emphasized using “o” 


6 Nonsmooth Data Fidelity 


» Figure 12 shows that there is a striking distinction in the behavior of the 
minimizers relevant to nonsmooth data-fidelity terms (b) with respect to 
nonsmooth regularization (a). More precisely, many data samples are fitted 
exactly when the data-fidelity term is nonsmooth. This particular behavior is 
explained and generalized in the present section. 


Consider 
F(u,v) = V(u, v) + B®(u), (29) 


q 
Yur) = Dov (au) — vill), (30) 


i=1 
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where a; € R” foralli € {1,...,g} andy : Ry — Ry} is a function satisfying 
H12 yw isC”, m > 2 onR*, and y' (0*) > 0 is finite. 

By this condition, t + y(|f|) is continuous on R. Let A € R4*? denote the 
matrix such that for any? = 1,...,g, its ith row reads ay. 


Nonsmooth data-fidelity terms Y in energies of the form (29) and (30) were 
introduced in image processing in 2001 [77]. 


General Results 

Here we present some results on the minimizers u of F as given in (29) and (30), 
where W is nondifferentiable, obtained in [78,79]. An additional assumption is that 
H13 The regularization term ® : R? > R in (29) isC”, m = 2. 


Note that ® in (29) can be convex or nonconvex. To analyze the observation in 
Fig. 12b, the following set-valued function 7 will be useful: 


(u,v) €(R?xR) & Juv) = {i E{1,...,g}: (aj,u) = util}. (31) 


Given v and a (local) minimizer # of F(-, v), the set of all data entries v[7] that are 
fitted exactly by (Au)[i] reads J = J(u, v). Its complement is J‘ = {1,...,q}\J. 


Theorem 11. Let F be of the form (29)-(30) where assumptions H12 and H13 
hold. Given v € R4, let i € R” be a (local) minimizer of F(-,v). For J = J(u, y), 
where J is defined according to (31), let 


Kj(v) = {ue R?: (a;,u) = vi] Vi € F and (a;,u) F vfi] Vi € Jt, 
and let K ; be its tangent. Suppose the following: 


1. The set {ai re ji is linearly independent. 


2. Vw € K; \ {0} we have DiF lize) H, v)w = Oand Di (Fla) i v)(w, w) > 
0. 
3. Wwe K; \ {0} we have 5,F (i, v)(w) > 0. 


Then there are a neighborhood O; C RY‘ containing v and a C”—! local minimizer 
function U : O; — R? relevant to F(-, O;), yielding in particular u = U(v), and 


(a;,U(v)) = vii] ifie J, 


(a;,U(W)) # vii] fie F*. “2 


veO; > 


The result in (32) means that 7(U(v), v) = J is constant on O;. 
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Note that for every v and J # @, the set K;(v) is a finite union of connected 
components, whereas its closure K’;(v) is an affine subspace. Its tangent K ; reads 


Kj ={ueR?: (a;,u) =0Vi € J}. 


A comparison with K; in (24) may be instructive. Compare also (b) and (c) in 
Theorem 11 with (a) and (b) in Theorem 10. By the way, conditions (b) and (c) in 
Theorem 11 ensure that F(-,v) reaches a strict minimum at i [78, Proposition 1]. 
Observe that this sufficient condition for strict minimum involves the behavior 
of F(-,v) on two orthogonal subspaces separately. This occurs because of the 
nonsmoothness of t +> w(|f|) at zero. It can be useful to note that at a minimizer i, 


51 F(a, vw) = $'(0*) 9) [Mai w)| + D2 w' (ai. a) — vli)ai,w) 
ies ie fe 


+BD ©(a)w = 0, for any w € R? (33) 


Commentary on the assumptions. Assumption (a) does not require the indepen- 
dence of the whole set {a; :i € {1,...,q}}. It is easy to check that this assumption 
fails to hold only for some v is included in a subspace of dimension strictly smaller 
than g. Hence, assumption (a) is satisfied for almost all v € R? and the theorem 
addresses any matrix A, whether it be singular or invertible. 

Assumption (b) is the classical sufficient condition for a strict local minimum of 
a smooth function over an affine subspace; see Lemma | (section “Reminders and 
Definitions”). If an arbitrary function F(-,v) : R? > R has a minimum at u, then 
necessarily 6;F (i, v)(w) = 0 for all w € K;; see Theorem |. In comparison, (c) 
requires only that the latter inequality be strict. 

It will be interesting to characterize the sets of data v for which (b) and (c) may 
fail at some (local) minimizers. Some ideas from section “Local Minimizers” can 
provide a starting point. 


Corollary 2. Let F be of the form (29)-(30) where p = q, and H12 and H13 hold 
true. Given v € RY¢, let i € R? be a (local) minimizer of F(-,v). Suppose the 
following: 


(a) The set {aj : 1 <i < q} is linearly independent. 


q 
(b) Vw € RY satisfying ||w\|2 = 1 we have B |D®(a)w| < w/(0T) > |(a;, w)|. 


i=l 
Then 
j= {1,...,g} 


and there are a neighborhood O; C IR‘ containing v and a C”—! local minimizer 
function U : O; > R? relevant to F(-, O;), yielding in particular i = U(v), and 
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veO; => (a;,U(v)) = vii] Vie J = fl,...,q}. (34) 


More precisely, U(v) = A~'v for any v € O;. 


In the context of Corollary 2, A is invertible. Combining this with (33) and (b) 
shows that 


K;(v) = {ue R?: Au=v}= Any: 
Kj; = ker(A) = {0}. 


Then 


q 
ve R?: B|D&(A™v)w| < WOT) > Kaj. w)|, Vw € R? \ {0}, |lwll2 = 1 


i=1 


The subset on the left contains an open subset of R’ by the continuity of v te 
D®(A~'y) combined with (b). 


Significance of the results. Consider that #J > 1. The result in (32) means that 
the set-valued function v > 7(U(v), v) is constant on O jz, Le., that 7 is constant 
under small perturbations of v. Equivalently, all residuals (a;,/(v)) —v[i] fori € - 
are null on O;. 

Theorem 11 shows that IR? contains volumes of positive measure composed of 
data that lead to local minimizers which fit exactly the data entries belonging to 
the same set. In general, there are volumes corresponding to various J so that 
noisy data come across them. That is why nonsmooth data-fidelity terms generically 
yield minimizers fitting exactly a certain number of the data entries. The resultant 
numerical effect is observed in Fig. 12b as well as in Figs. 14 and 15. 


Remark 3 (Stability of Minimizers). The fact that there is a C’”—! local minimizer 
function shows that, in spite of the nonsmoothness of ¥, for any v, all local 
minimizers of F(-,v) which satisfy the conditions of the theorem are stable under 
weak perturbations of data v. This result extends Lemma |. 
Example 6. Let F read 
q B q ‘ 
Fu,v) =) luli] — vill + 5D) Hl)”. B > 0. 


i=1 i=1 


It is easy to see that there is a unique local minimizer function / which is given by 
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Fig. 13 Original u, and data v degraded by outliers. (a) Original u,. (b) Data v = u- outliers 


U(v)[i] = 5 santé) if |v[z]| > 


U(v)[i] = v[i] if |vfi]| < 


’ 


Tile Wile 


Condition (c) in Theorem 11 fails to hold only for \v ER: |i]= 4, vies \. 
This set is of Lebesgue measure zero in R?. For any J € {1,...,q} put 


1 
O; =3veER?‘: |vfi]] < —, Vie J and well > Ze wie st 


1 

B 

Obviously, every v € O, gives rise to a minimizer a satisfying 
ui] = v[i], Vie J and afi] A vfi], Vie J°. 


Each set O,; has a positive Lebesgue measure in R?. Moreover, the union of all O; 
when J ranges on all subsets J C {1,...,q} (including the empty set) forms a 
partition of R‘. 


Numerical experiment. The original image u, is shown in Fig. 13a. Data v in 
Fig. 13b are obtained by replacing some pixels of u, by aberrant impulsions, called 
outliers. 

In all Figs. 14-17, {D;} correspond to the first-order differences between each 
pixel and its four nearest neighbors. Figure 14a corresponds to an £, data-fidelity 
term for B = 0.14. The outliers are well visible although their amplitudes are clearly 
reduced. The image of the residuals v — i, shown in Fig. 14b, is null everywhere 
except at the positions of the outliers in v. The pixels corresponding to nonzero 
residuals (i.e., the elements of J “) provide a good estimate of the locations of the 
outliers in v. Figure 15a shows a minimizer u of the same F(-,v) obtained for 
B = 0.25. This minimizer does not contain visible outliers and is very close to 
the original image u,. The image of the residuals v — a in Fig. 15b is null only on 
restricted areas but has a very small magnitude everywhere beyond the outliers. 
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Fig. 14 Restoration using F(u,v) = >; |u[i] — v[f]| + B Vie, |Diul” w = 1.1 and B = 0.14. 
(a) Restoration u for B = 0.14. (b) Residual v — a 


Fig. 15 Restoration using F(u, v) = >°; |ufi]—v[i]| +6 >0;<, |Dul* fora = 1.1 and B = 0.25. 
(a) Restoration u for B = 0.25. (b) Residual v — a 


Fig. 16 Restoration using a smooth energy, F(u, v) = >>; (u[i]—v[i])? +B ©; ((Diul)*, B = 0.2. 
(a) Restoration u for B = 0.2. (b) Residual v — x 
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Fig. 17 Restoration using nonsmooth regularization F(u, v) = Yuli) —v[i]? + B ye |D;ul, 


B = 0.2. (a) Restoration w for B = 0.2. (b) Residual v — a 


The minimizers of two different cost-functions F involving a smooth data- 
fidelity term W, shown in Figs. 16 and 17, do not fit any data entry. In particular, the 
restoration in Fig. 17 corresponds to a nonsmooth regularization and it is constant 
over large regions; this effect was explained in Sect. 5. 
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Applications 


The possibility to keep some data samples unchanged by using nonsmooth data 
fidelity is a precious property in various application fields. Nonsmooth data fidelities 
are good to detect and smooth outliers. This property was exploited for deblurring 
under impulse noise contamination; see, e.g., [10-12]. 


Denoising of frame coefficients. Consider the recovery of an original (unknown) 
Uy € R? — a signal or an image containing smooth zones and edges — from noisy 
data 


V=Uo +n, 


where 7 represents a perturbation. As discussed in Sect. 4, a systematic default of 
the images and signals restored using convex edge-preserving PFs ¢ is that the 
amplitude of edges is underestimated. 

Shrinkage estimators operate on a decomposition of data v into a frame of £7, 
say {w; : i € J} where J is a set of indexes. Let W be the corresponding frame 
operator, i.e., (Wv)[i] = (v,w;), Vi € J, and W be a left inverse of W, giving 
rise to the dual frame {w; : i € J}. The frame coefficients of v read y = Wv and 
are contaminated with noise Wn. The inaugural work of Donoho and Johnstone 
[40] considers two different shrinkage estimators: given T > 0, hard thresholding 
corresponds to 


yli] ifi € Jj, ee Jo={ieJ: |y[i]| < T}: 


35 
0 ifi € Jo, J, = J\Jo, ( ) 


yrli] = 


while in soft thresholding one takes yr[i] = y[i] — Tsign(y[i]) if i € J, and 
yr|i] = Oif i € Jo. Both soft and hard thresholding are asymptotically optimal in 
the minimax sense if 7 is white Gaussian noise of standard deviation o and 


T =o ,/2log, p. (36) 


This threshold is difficult to use in practice because it increases with the size 
of u. Numerous improvements were realized; see, e.g., [4, 13, 24, 34, 38, 66, 70]. 
In all cases, the main problem is that smoothing large coefficients oversmooths 
edges, while thresholding small coefficients can generate Gibbs-like oscillations 
near edges; see Fig. 18c, d. If shrinkage is weak, noisy coefficients (outliers) remain 
almost unchanged and produce artifacts having the shape of {w;}; see Fig. 18c-e. 

In order to alleviate these difficulties, several authors proposed hybrid methods 
where the information contained in important coefficients y[i] is combined with 
priors in the domain of the sought-after signal or image [18,25, 36,43, 67]. A critical 
analysis was presented in [46]. 

A specialized hybrid method involving £; data fidelity on frame coefficients is 
proposed in [46]. Data are initially hard thresholded — see (35) — using a suboptimal 
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Fig. 18 Methods to restore the noisy signal in (a). Restored signal (—), original signal (- -). (a) 
Original and data corrupted with white Gaussian noise. (b) TV regularization. (c) Sure-shrink. (d) 
T = 35 optimal, ap = °; yr[i]w;. (©) yr, T = 23, up = >; yr[i]w;. ) The proposed method 


threshold T in order to keep as much as possible information. (The use of another 
shrinkage estimator would alter all coefficients, which is not desired.) Then 


1. J, is composed of: 


¢ Large coefficients bearing the main features of u, that one wishes to preserve 
intact 

e Aberrant coefficients (outliers) that must be restored using the regularization 
term 


2. Jo is composed of: 


¢ Noise coefficients that must be kept null. 

* Coefficients y[i] corresponding to edges and other details in u, — these need 
to be restored in accordance with the prior incorporated in the regularization 
term. 


In order to reach the goals formulated in 1 and 2 above, denoised coefficients X are 
defined as a minimizer of the hybrid energy F(., y) given below: 


F(x, y) = Ar > [xfé] — yl] + Ao SS [xf + 92 ¢ (IDI Wxll2), Ao > 0, 
1éJ, 1EJo ie] 
(37) 


where @ is convex and edge preserving. Then the sought-after denoised image or 
signal is 


194 M. Nikolova 


Several properties relevant to the minimizers of F in (37), the parameters A;,i € 
{0, 1}, and the solution @ are outlined in [46]. 

Noisy data v are shown along with the original u, in Fig. 18a. The restoration in 
Fig. 18b minimizes F(u) = ||Au — v||} + BTV — homogeneous regions remain 
noisy, edges are smoothed, and spikes are eroded. Figure 18c is obtained using 
the sure-shrink method [41] from the toolbox WaveLab. The other restorations use 
thresholded Daubechies wavelet coefficients with eight vanishing moments. The 
optimal value for the hard thresholding obtained using (36) is T = 35. The relevant 
restoration — Fig. 18d — exhibits important Gibbs-like oscillations as well as wavelet- 
shaped artifacts. For T = 23 the coefficients have a richer information content, 
but W yr, shown in Fig. 18e, manifests Gibbs artifacts and many wavelet-shaped 
artifacts. Introducing the thresholded coefficients of Fig. 18e in the specialized 
energy F in (37) leads to Fig. 18f: edges are clean and piecewise polynomial parts 
are well recovered. 


7 Nonsmooth Data Fidelity and Regularization 
The L,-TV Case 


For discrete signals of finite length, energies of the form F(u,v) = ||u —v|l1 + 
pyr |u[i + 1] — u[i]| were considered by Alliney in 1992 [1]. 
Following [1, 4 79], S. Esedoglu and T. Chan explored in [28] the minimizers 
of the L,-TV functional given below 


Fl(u,v) = [, |u(x) — v(x)|dx + pf |Vu(x)|dx, (38) 


where the sought-after minimizer % belongs to the space of bounded variation 
functions on R¢, The main focus is on images, i.e, d = 2. The analysis in [28] 
is based on a representation of F in (38) in terms of the level sets of u and v. Most 
of the results are established for data v given by the characteristic function 7» of a 
bounded domain © C R¢. Theorem 5.2 in [28] says that if v = ys, where © C R¢ 
is bounded, then F(-,v) admits a minimizer of the form u = x» (with possibly 
p> # %D). Furthermore, Corollary 5.3. in [28] states that if in addition © is convex, 
then for almost every 8 > 0, F(-,v) admits a unique minimizer and w# = y¢ 
with 5 Cc &. Moreover, it is shown that small features in the image maintain 
their contrast intact up to some value of 6, while for a larger 6 they suddenly 
disappear. 
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Noisy binary Restored binary 


Fig. 19 Restoration of a binary noisy image by minimizing L,-TV 


Denoising of Binary Images and Convex Relaxation 
Many problems such as text denoising and document processing, two-phase image 
segmentation, shape restoration, and fairing of surfaces in computer graphics are 
naturally stated as the minimization of an energy over the set of the binary images. 
These energies are obviously nonconvex since the constraint set is finite. Their 
global minimizer was shown in [29] to be also the minimizer of the convex L- 
TV functional which is convex. This result yielded much simpler algorithms for 
binary image restoration. An illustration is given on Fig. 19. 

Since then, L,-TV relaxations have became a common tool for convex relax- 
ations; see, e.g., among many others [84] and the references therein. 

Also, L,-TV energies were revealed very successful in image decomposition; 
see, e.g., [8,48]. 


Multiplicative Noise Removal 

In various active imaging systems, such as synthetic aperture radar (SAR), laser, or 
ultrasound imaging, the data representing the underlying (unknown image) So are 
corrupted with multiplicative noise. Such a noise is a severe degradation; see Fig. 20. 
When possible, a few independent measurements for the same scene are realized, 
Xe = Son, fork € {1,...,K}, where the noise x is typically modeled by the 
one-sided exponential distribution. The data © used for denoising is the average of 
the set of all K measurements: 


K 
1 
== KX De = Son. (39) 


The combined multiplicative noise follows a Gamma distribution. Adaptive filtering 
works only if the noise is weak. For strong noise, variational methods often use TV 
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Noisy: K=4in (39) [6]: PSNR=24.55, MAE=10.06 


[47]: PSNR=25.84, MAE=9.09 Original(512*512) 


Fig. 20 Aerial image of the town of Nimes (512 x 512) for K = 4 in (39). Restorations using 
different methods. Parameters: [6] for o = 120; [47] T = 2./W1(K), Ao = 1.5,A1 = 10 


regularization. In [6] the log-likelihood of the raw data (39) is regularized using TV. 
Instead, the properties of L,-TV are used to design an energy in [47]. First, the log- 
data v = log & is decomposed into a curvelet transform yielding noisy coefficients 
y = Wy. A suboptimal hard thresholding is applied for T adapted to the expectation 
of the log noise. Let J) = {i : |y[i] < T}and J, = {i : |y[i] > T}. Since the 
threshold is low, /, contains outliers. Coefficients * are restored by minimizing 


F(x) =A1 D0 |e = yf] + A0 DS [xf] + TV(X). 
ie€l, i€ly 


The restored image S, shown in Fig. 20, is obtained as S= exp(W (%)) B where W 
is a left inverse of W and B is a bias correction. 
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Fig. 21 Minimizers of F(-, v) as given in (40) for #(¢) = In(at + 1), a = 2 and different values 
of B. Data samples (0 © ©), minimizer samples u[i] (+ + +) 
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Fig. 22 Minimizers of F(-,v) as given in (40) for different PFs @. Data are corrupted with 
Gaussian noise. Data samples v[i] are marked with (0 © ©), samples u[i] of the minimizer — with 
(+ + +). The original signal is reminded in (— — —). (a) d(t) = rene a= 4, B = 3. (b) 
o(t)=t, B =0.8 


£, Data Fidelity with Regularization Concave on R4+ 


One could expect that £; data fidelity regularized with a PF concave on R+ should 
somehow reinforce the properties of £; — TV. The question was recently examined 
in [83]. Consider the energy 


F(u,v) = > |(ai,u) — vill +B >) o((Diul) (40) 


i€l jet 
for r= 41,...,9} and Sa... 


where @ : Ry — R¥ is continuous and concave on R+ (e.g., (£10), (f11), and (f12) 
in Table 1). 


Motivation 
Figures 21 and 22 depict (the global) minimizers of F(u,v) in (40) for a one- 
dimensional signal where A = Id, {D;} are first-order differences, and ¢ is smooth 
and concave on R_. 

The tests in Figs.21 and 22 show that a PF concave on R+ considerably 
reinforces the properties of £;—-TV. One observes that the minimizer satisfies exactly 
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part of the data term and part of the prior term (corresponding to constant pieces). 
In Fig. 22b, the previous £; — TV model is considered. Figure 21 shows also that 
the minimizer remains unchanged for some range of values of f and that after a 
threshold value, it is simplified. 

Example 7 below furnishes a first intuition on the reasons underlying the 
phenomena observed in Figs. 21 and 22. 


Example 7. Given v € R, consider the function F(-,v) : R — R given below 
F(u,v) = |u—v| + Bo(\u|) for @ obeying H. (41) 


The necessary conditions for F(-, v) to have a (local) minimum at # 4 0 anda 4 v 
— that its first differential meets D, F(a, v) = 0 and that its second differential obeys 
Di F (it, v) = 0 — do not hold: 


DF (i, v) = sign(u — v) + Be’ (\a|)sign(@@) = 0, 
id{0vi = 
Di F (i, v) = By" (lal) < 0, 


where the last inequality comes from the strict concavity of @ on R7_. Hence, F(-, v) 
cannot have a minimizer such that u 4 0 and tu # v, for any v € R. Being coercive, 
F(-,v) does have minimizers. Consequently, any (local) minimizer of F in (41) 
satisfies 


i € {0,v}. 


Main Theoretical Results 

The PFs considered here are concave on Ry and smooth on R*,. More precisely, 
they satisfy H1 (Sect. 1), H8, and H10 (section “Assumptions on Potential Functions 
g’’). One can see Fig. 3, right, for an illustration of the assumptions. 


Proposition 3. Let F(-,v) read as in (40). Assume that H3 (Sect. 3) holds and that 
@ satisfies HI (Sect. 1), H8, and H10. Then for any v, F(-, v) has global minimizers. 


Given v € R’, with each i € R” the following subsets are associated: 


fy @ fi eT | (aj) =vii}} and fe 27 \ hy, 13) 
jf fier | Da=0} and JB I\ A. 


Proposition 4. For F(-, v) as in (40) satisfying H1, H8, and H10, let i be a (local) 
minimizer of F(-,v). Then 


(fo U Jo) AS. 
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H14 The point u € R? is such that hh # @ and that 
w €kerD \ {0} = Fi € Io such that (a;,w) £0. (43) 
If rank D = p, then (43) is trivial. Anyway, (43) is not a strong requirement. 


Theorem 12. Consider F(-,v), as given in (40), satisfying H3, as well as H1, H8, 
and H10. Let it be a (local) minimizer of F(-,v) meeting Ji # @ and H14. Then i 
is the unique solution of the full column rank linear system given below 


(a;,w) = vii] is Io , (44) 
Djw=0 Vj EJ. 

Significance of the Results 

An immediate consequence of Theorem 12 is the following: 


em Each (local) minimizer of F(-, v) is strict. 


Another consequence is that the matrix H with rows (a; ,Vi € fy and D pVvye€ 
Jo) has full column rank. This provides a strong necessary condition for a (local) 
minimizer of F(-,v). And since i in (44) solves a linear system, it involves the 
same kind of “contrast invariance” as the L; — TV model. A detailed inspection 
of the minimizers in Figs.21 and 22 corroborate Theorem 12. A more practical 
interpretation of this result reads as follows: 


> Each pixel of a (local) minimizer a of F(-,v) is involved in (at least) one 
data equation that is fitted exactly (a;,a) = v[i] or in (at least) one vanishing 
operator D; i = 0 or in both types of equations. 


This remarkable property can be used in different ways. 


Applications 
An energy F(-,v) of the form in (40) with a PF ¢ strictly concave on R+ is a good 
choice when 


¢ There are some nearly faithful data points v[7]; 
e The matrix D provides a very reliable prior on the sought-after solution. 


A natural way for such a prior is to construct for D an application-dependent 
dictionary. 


MR Image Reconstruction from Highly Undersampled Data 
In the experiment in Fig. 23, only 5 % randomly chosen noisy data samples in the 
k-space (i.e., individual noisy Fourier coefficients) are available; see (a). Data are 
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Fig. 23 Reconstructed images from 5 % noisy randomly selected samples in the k-space using 
different methods. (a) Zero-filling Fourier recovery. (b) £2 — TV. (c) F(-, v) in (40) 


contaminated with SNR=37dB white centered Gaussian noise. This is a highly 
underdetermined, ill-posed inverse problem. It can be related to compressed sensing 
in MRI; see, e.g., [58]. The Shepp-Logan phantom being locally constant with oval 
shapes, the linear operators {D;} in (40) yield the usual discrete gradient of the 
image, so that the regularization term provides a correct prior. Indeed, Du, is the 
sparsest linear transform for this image. Clearly, A is the undersampled Fourier 
transform corresponding to the 5% randomly chosen k-samples. For Gaussian 
noise, an £3 quadratic data fitting term is a classical choice. The £2 — TV cost- 
function || Au — v||} + BTV(u) is the standard tool to solve this kind of problems. 
The result is shown in Fig. 23b. 


8 Conclusion 


This chapter provided some theoretical results relating the shape of the energy F 
to minimize and the salient features of its minimizers @ (see (7), section “The Main 
Features of the Minimizers as a Function of the Energy’’). These results can serve as 
a kind of inverse modeling: given an inverse problem along with our requirements 
(priors) on its solution, they guide us how to construct an energy functional whose 
minimizers properly incorporate all this information. The theoretical results are 
illustrated using numerical examples. Various application fields can take a benefit 
from these results. The problem of such an inverse modeling remains open because 
of the diversity of the inverse problems to solve and the possible energy functionals. 


Cross-References 


> Compressive Sensing 

> Inverse Scattering 

> Iterative Solution Methods 
> Linear Inverse Problems 
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Mathematical Methods in PET and SPECT Imaging 

Mumford and Shah Model and Its Applications to Image Segmentation and Image 
Restoration 

Regularization Methods for Ill-Posed Problems 

Statistical Methods in Imaging 

Total Variation in Imaging 
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Abstract 


Compressive sensing is a recent type of sampling theory, which predicts that 
sparse signals and images can be reconstructed from what was previously 
believed to be incomplete information. As a main feature, efficient algorithms 
such as £;-minimization can be used for recovery. The theory has many potential 
applications in signal processing and imaging. This chapter gives an introduction 
and overview on both theoretical and numerical aspects of compressive sensing. 


1 Introduction 


The traditional approach of reconstructing signals or images from measured data 
follows the well-known Shannon sampling theorem [155], which states that the 
sampling rate must be twice the highest frequency. Similarly, the fundamental theo- 
rem of linear algebra suggests that the number of collected samples (measurements) 
of a discrete finite-dimensional signal should be at least as large as its length (its 
dimension) in order to ensure reconstruction. This principle underlies most devices 
of current technology, such as analog to digital conversion, medical imaging, or 
audio and video electronics. The novel theory of compressive sensing (CS) — also 
known under the terminology of compressed sensing, compressive sampling, or 
sparse recovery — provides a fundamentally new approach to data acquisition which 
overcomes this common wisdom. It predicts that certain signals or images can be 
recovered from what was previously believed to be highly incomplete measurements 
(information). This chapter gives an introduction to this new field. Both fundamental 
theoretical and algorithmic aspects are presented, with the awareness that it is 
impossible to retrace in a few pages all the current developments of this field, which 
was growing very rapidly in the past few years and undergoes significant advances 
on an almost daily basis. 

CS relies on the empirical observation that many types of signals or images can 
be well approximated by a sparse expansion in terms of a suitable basis, that is, by 
only a small number of nonzero coefficients. This is the key to the efficiency of many 
lossy compression techniques such as JPEG, MP3, etc. A compression is obtained 
by simply storing only the largest basis coefficients. When reconstructing the signal, 
the non-stored coefficients are simply set to zero. This is certainly a reasonable 
strategy when full information of the signal is available. However, when the signal 
first has to be acquired by a somewhat costly, lengthy, or otherwise difficult 
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measurement (sensing) procedure, this seems to be a waste of resources: First, large 
efforts are spent in order to obtain full information on the signal, and afterwards 
most of the information is thrown away at the compression stage. One might ask 
whether there is a clever way of obtaining the compressed version of the signal 
more directly, by taking only a small number of measurements of the signal. It is not 
obvious at all whether this is possible since measuring directly the large coefficients 
requires to know a priori their location. Quite surprisingly, compressive sensing 
provides nevertheless a way of reconstructing a compressed version of the original 
signal by taking only a small amount of linear and nonadaptive measurements. 
The precise number of required measurements is comparable to the compressed 
size of the signal. Clearly, the measurements have to be suitably designed. It is a 
remarkable fact that all provably good measurement matrices designed so far are 
random matrices. It is for this reason that the theory of compressive sensing uses a 
lot of tools from probability theory. 

It is another important feature of compressive sensing that practical recon- 
struction can be performed by using efficient algorithms. Since the interest is 
in the vastly undersampled case, the linear system describing the measurements 
is underdetermined and therefore has infinitely many solutions. The key idea is 
that the sparsity helps in isolating the original vector. The first naive approach 
to a reconstruction algorithm consists in searching for the sparsest vector that is 
consistent with the linear measurements. This leads to the combinatorial £9-problem 
(see (4) below), which unfortunately is NP-hard in general. There are essentially 
two approaches for tractable alternative algorithms. The first is convex relaxation 
leading to £;-minimization — also known as basis pursuit (see (5)) — while the 
second constructs greedy algorithms. This overview focuses on £;-minimization. 
By now basic properties of the measurement matrix which ensure sparse recovery 
by €,-minimization are known: the null space property (NSP) and the restricted 
isometry property (RIP). The latter requires that all column submatrices of a certain 
size of the measurement matrix are well conditioned. This is where probabilistic 
methods come into play because it is quite hard to analyze these properties for 
deterministic matrices with minimal amount of measurements. Among the provably 
good measurement matrices are Gaussian, Bernoulli random matrices, and partial 
random Fourier matrices. 

Figure | serves as a first illustration of the power of compressive sensing. It 
shows an example for recovery of a 10-sparse signal x € C*°? from only 30 samples 
(indicated by the red dots in Fig. 1b). From a first look at the time-domain signal, one 
would rather believe that reconstruction should be impossible from only 30 samples. 
Indeed, the spectrum reconstructed by traditional £-minimization is very different 
from the true spectrum. Quite surprisingly, €;-minimization performs nevertheless 
an exact reconstruction, that is, with no recovery error at all! 

An example from nuclear magnetic resonance imaging serves as a second illus- 
tration. Here, the device scans a patient by taking 2D or 3D frequency measurements 
within a radial geometry. Figure 2a describes such a sampling set of a 2D Fourier 
transform. Since a lengthy scanning procedure is very uncomfortable for the patient, 
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Fig. 1 (a) 10-sparse Fourier spectrum, (b) time-domain signal of length 300 with 30 samples, (c) 
reconstruction via £)-minimization, (d) exact reconstruction via ¢;-minimization 


it is desired to take only a minimal amount of measurements. Total variation 
minimization, which is closely related to £,-minimization, is then considered as 
recovery method. For comparison, Fig.2b shows the recovery by a traditional 
back-projection algorithm. Figure 2c, d displays iterations of an algorithm, which 
was proposed and analyzed in [72] to perform efficient large-scale total variation 
minimization. The reconstruction in Fig. 2d is again exact! 


2 Background 


Although the term compressed sensing (compressive sensing) was coined only 
recently with the paper by Donoho [47], followed by a huge research activity, 
such a development did not start out of thin air. There were certain roots and 
predecessors in application areas such as image processing, geophysics, medical 
imaging, computer science, as well as in pure mathematics. An attempt is made 
to put such roots and current developments into context below, although only a 
partial overview can be given due to the numerous and diverse connections and 
developments. 
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Fig. 2 (a) Sampling data of the NMR image in the Fourier domain which corresponds to only 
0.11% of all samples. (b) Reconstruction by back projection. (c) Intermediate iteration of an 
efficient algorithm for large-scale total variation minimization. (d) The final reconstruction is exact 


Early Developments in Applications 


Presumably the first algorithm which can be connected to sparse recovery is 
due to the French mathematician de Prony [127]. The so-called Prony method, 
which has found numerous applications [109], estimates nonzero amplitudes and 
corresponding frequencies of a sparse trigonometric polynomial from a small 
number of equispaced samples by solving an eigenvalue problem. The use of £)- 
minimization appears already in the Ph.D. thesis of B. Logan [106] in connection 
with sparse frequency estimation, where he observed that L;-minimization may 
recover exactly a frequency-sparse signal from undersampled data provided the 
sparsity is small enough. The paper by Donoho and Logan [52] is perhaps the 
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earliest theoretical work on sparse recovery using L,-minimization. Nevertheless, 
geophysicists observed in the late 1970s and 1980s that £;-minimization can be 
successfully employed in reflection seismology where a sparse reflection function 
indicating changes between subsurface layers is sought [140, 148]. In NMR 
spectroscopy the idea to recover sparse Fourier spectra from undersampled non- 
equispaced samples was first introduced in the 1990s [158] and has seen a significant 
development since then. 

In image processing the use of total variation minimization, which is closely 
connected to £;-minimization and compressive sensing, first appears in the 1990s 
in the work of Rudin, Osher, and Fatemi [139] and was widely applied later on. 
In statistics where the corresponding area is usually called model selection, the use 
of £,-minimization and related methods was greatly popularized with the work of 
Tibshirani [149] on the so-called LASSO (Least Absolute Shrinkage and Selection 
Operator). 


Sparse Approximation 


Many lossy compression techniques such as JPEG, JPEG-2000, MPEG, or MP3 
rely on the empirical observation that audio signals and digital images have a sparse 
representation in terms of a suitable basis. Roughly speaking one compresses the 
signal by simply keeping only the largest coefficients. In certain scenarios such 
as audio signal processing, one considers the generalized situation where sparsity 
appears in terms of a redundant system — a so-called dictionary or frame [36] — rather 
than a basis. The problem of finding the sparsest representation/approximation in 
terms of the given dictionary turns out to be significantly harder than in the case 
of sparsity with respect to a basis where the expansion coefficients are unique. 
Indeed, in [108, 114], it was shown that the general fo-problem of finding the 
sparsest solution of an underdetermined system is NP-hard. Greedy strategies such 
as matching pursuit algorithms [108], FOCUSS [86] and £,-minimization [35] were 
subsequently introduced as tractable alternatives. The theoretical understanding 
under which conditions greedy methods and £\-minimization recover the sparsest 
solutions began to develop with the work in [50,51, 62, 78, 81,87, 151, 152]. 


Information-Based Complexity and Gelfand Widths 


Information-based complexity (IBC) considers the general question of how well 
a function f belonging to a certain class F can be recovered from n sample 
values or, more generally, the evaluation of n linear or nonlinear functionals 
applied to f [150]. The optimal recovery error which is defined as the maximal 
reconstruction error for the “best” sampling method and “best” recovery method 
(within a specified class of methods) over all functions in the class F is closely 
related to the so-called Gelfand width of F [38,47, 117]. Of particular interest for 
compressive sensing is F = BN , the £;-ball in R” since its elements can be well 
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approximated by sparse ones. A famous result due to Kashin [96] and Gluskin and 
Garnaev [79, 84] sharply bounds the Gelfand widths of BN (as well as their duals, 
the Kolmogorov widths) from above and below; see also [77]. While the original 
interest of Kashin was in the estimate of n-widths of Sobolev classes, these results 
give precise performance bounds in compressive sensing on how well any method 
may recover (approximately) sparse vectors from linear measurements [38,47]. The 
upper bounds on Gelfand widths were derived in [96] and [79] using (Bernoulli and 
Gaussian) random matrices (see also [107]), and in fact such type of matrices have 
become very useful also in compressive sensing [26,47]. 


Compressive Sensing 


The numerous developments in compressive sensing began with the seminal work 
[30,47]. Although key ingredients were already in the air at that time, as mentioned 
above, the major contribution of these papers was to realize that one can combine 
the power of €)-minimization and random matrices in order to show optimal 
results on the ability of £;-minimization of recovering (approximately) sparse 
vectors. Moreover, the authors made very clear that such ideas have strong potential 
for numerous application areas. In their work [26, 30] Candés, Romberg, and 
Tao introduced the restricted isometry property (which they initially called the 
uniform uncertainty principle) which is a key property of compressive sensing 
matrices. It was shown that Gaussian, Bernoulli, and partial random Fourier matrices 
[26, 129, 138] possess this important property. These results require many tools from 
probability theory and finite-dimensional Banach space geometry, which have been 
developed for a rather long time now; see, e.g., [95, 103]. 

Donoho [49] developed a different path and approached the problem of charac- 
terizing sparse recovery by ¢\-minimization via polytope geometry, more precisely, 
via the notion of k-neighborliness. In several papers sharp phase transition curves 
were shown for Gaussian random matrices separating regions where recovery fails 
or succeeds with high probability [49, 53,54]. These results build on previous work 
in pure mathematics by Affentranger and Schneider [2] on randomly projected 
polytopes. 


Developments in Computer Science 


In computer science the related area is usually addressed as the heavy hitters 
detection or sketching. Here one is interested not only in recovering signals (such as 
huge data streams on the Internet) from vastly undersampled data, but one requires 
sublinear runtime in the signal length N of the recovery algorithm. This is no 
impossibility as one only has to report the locations and values of the nonzero (most 
significant) coefficients of the sparse vector. Quite remarkably sublinear algorithms 
are available for sparse Fourier recovery [80]. Such algorithms use ideas from group 
testing which date back to World War II, when Dorfman [56] invented an efficient 
method for detecting draftees with syphilis. 
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In sketching algorithms from computer science, one actually designs the matrix 
and the fast algorithm simultaneously [42, 82]. More recently, bipartite expander 
graphs have been successfully used in order to construct good compressed sensing 
matrices together with associated fast reconstruction algorithms [11]. 


3 Mathematical Modelling and Analysis 


This section introduces the concept of sparsity and the recovery of sparse vectors 
from incomplete linear and nonadaptive measurements. In particular, an analysis of 
£,-minimization as a recovery method is provided. The null space property and the 
restricted isometry property are introduced, and it is shown that they ensure robust 
sparse recovery. It is actually difficult to show these properties for deterministic 
matrices and the optimal number m of measurements, and the major breakthrough 
in compressive sensing results is obtained for random matrices. Examples of several 
types of random matrices which ensure sparse recovery are given, such as Gaussian, 
Bernoulli, and partial random Fourier matrices. 


Preliminaries and Notation 


This exposition mostly treats complex vectors in C’ although sometimes the 
considerations will be restricted to the real case R”. The p-horm of a vector 
x € C% is defined as 


x 1/p 
IIx llp := Siar ;, 0< p<, 
j= 
IX lloo = Fae |x;|.- (1) 


For 1 < p < ov, it is indeed a norm, while for 0 < p < 1, it is only a quasi-norm. 
When emphasizing the norm, the term is used instead of C’ or R%. The unit 
ball in a is By = {x € CN, ||x||, < 1}. The operator norm of a matrix A € C”"*" 
from ie to £7) is denoted 


|Allp>p = anes || Axl. (2) 


X||[p=1 


In the important special case p = 2, the operator norm is the maximal singular 
value Omax(A) of A. 

For a subset T C {1,...,N}, one denotes by xr € C% the vector which 
coincides with x € C% on the entries in T and is zero outside T. Similarly, Ar 
denotes the column submatrix of A corresponding to the columns indexed by T. 
Further, 7° = {1,..., N}\ 7 denotes the complement of T and #T or |T'| indicates 
the cardinality of T. The kernel of a matrix A is denoted by ker A = {x, Ax = O}. 
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Sparsity and Compression 


Compressive sensing is based on the empirical observation that many types of real- 

world signals and images have a sparse expansion in terms of a suitable basis or 

frame, for instance, a wavelet expansion. This means that the expansion has only a 

small number of significant terms, or, in other words, that the coefficient vector can 

be well approximated with one having only a small number of nonvanishing entries. 
The support of a vector x is denoted supp(x) = {j : x; 4 O} and 


|x [lo -= | supp(x)]. 


It has become common to call || - ||o the 2o-norm, although it is not even a quasi- 
norm. A vector x is called k-sparse if ||x||o < k. Fork € {1,2,..., N}, 


de = {x EC: |[xllo <3 


denotes the set of k-sparse vectors. Furthermore, the best k-term approximation 
error of a vector x € C% in ¢, is defined as 


OK(X)p = 


= anh Ix — zllp- 
If ox (x) decays quickly in k, then x is called compressible. Indeed, in order to 
compress x, one may simply store only the k largest entries. When reconstructing 
x from its compressed version, the non-stored entries are simply set to zero, and the 
reconstruction error is 0; (x). It is emphasized at this point that the procedure of 
obtaining the compressed version of x is adaptive and nonlinear since it requires 
the search of the largest entries of x in absolute value. In particular, the location of 
the nonzeros is a nonlinear type of information. 

The best k-term approximation of x can be obtained using the nonincreasing 


rearrangement r(x) = (|xj,|,...,|Xiy |)’, where i; denotes a permutation of the 
indexes such that |.x;;| > |xi,,,| for 7 = 1,..., NW — 1. Then it is straightforward to 
check that 
re \/p 
OK(X) p t= > rj (x)? , O<p<m. 
jak+l 


And the vector xx; derived from x by setting to zero all the N — k smallest entries 
in absolute value is the best k-term approximation, 


Xk) = arg min ||x — Zl|,, 
[kl g min | Il p 


forany0 < p< om. 
The next lemma states essentially that €,-balls with small g (ideally g < 1) are 
good models for compressible vectors. 
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Lemma 1. Let0 <q < p < wandsetr = . Then 


Idi 
q Pp 


GOp,ek >: KSI 2 aN foralxe Be, 


Proof. Let T be the set of indexes of the k-largest entries of x in absolute value. The 
nonincreasing rearrangement satisfies |r, (x)| < |x;| for all j € T, and therefore 


kri(x)? < >) |x;/? < llxlld <1. 
jeT 


Hence, rz(x) < koa. Therefore, 


= Pad _ P74 
ox)? = > xsl? < Dore)? Axl? Ske x2 kOe, 
JE¢T J¢T 


which implies 0; (x), < k~". | 


Compressive Sensing 


The above outlined adaptive strategy of compressing a signal x by only keeping 
its largest coefficients is certainly valid when full information on x is available. If, 
however, the signal first has to be acquired or measured by a somewhat costly or 
lengthy procedure, then this seems to be a waste of resources: At first, large efforts 
are made to acquire the full signal and then most of the information is thrown away 
when compressing it. One may ask whether it is possible to obtain more directly 
a compressed version of the signal by taking only a small amount of linear and 
nonadaptive measurements. Since one does not know a priori the large coefficients, 
this seems a daunting task at first sight. Quite surprisingly, compressive sensing 
nevertheless predicts that reconstruction from vastly undersampled nonadaptive 
measurements is possible — even by using efficient recovery algorithms. 

Taking m linear measurements of a signal x €¢ C% corresponds to applying a 
matrix A € C”™* — the measurement matrix — 


y = Ax. (3) 


The vector y € C” is called the measurement vector. The main interest is in the 
vastly undersampled case m < N. Without further information, it is, of course, 
impossible to recover x from y since the linear system (3) is highly underdetermined 
and has therefore infinitely many solutions. However, if the additional assumption 
that the vector x is k-sparse is imposed, then the situation dramatically changes as 
will be outlined. 

The approach for a recovery procedure that probably comes first to mind is to 
search for the sparsest vector x which is consistent with the measurement vector 
y = Ax. This leads to solving the €9-minimization problem 
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min ||z||o subject to Az = y. (4) 


Unfortunately, this combinatorial minimization problem is NP-hard in general 
[108, 114]. In other words, an algorithm that solves (4) for any matrix A and any 
right-hand side y is necessarily computationally intractable. Therefore, essentially 
two practical and tractable alternatives to (4) have been proposed in the literature: 
convex relaxation leading to £;-minimization — also called basis pursuit [35] — and 
greedy algorithms, such as various matching pursuits [151, 153]. Quite surprisingly 
for both types of approaches, various recovery results are available, which provide 
conditions on the matrix A and on the sparsity || ||) such that the recovered solution 
coincides with the original x and consequently also with the solution of (4). This 
is no contradiction to the NP-hardness of (4) since these results apply only to a 
subclass of matrices A and right-hand sides y. 
The £;-minimization approach considers the solution of 


min ||z||; | subject to Az = y, (5) 


which is a convex optimization problem and can be seen as a convex relaxation 
of (4). Various efficient convex optimization techniques apply for its solution [17]. 
In the real-valued case, (5) is equivalent to a linear program, and in the complex- 
valued case, it is equivalent to a second-order cone program. Therefore, standard 
software applies for its solution — although algorithms which are specialized to (5) 
outperform such standard software; see Sect. 4. 

The hope is, of course, that the solution of (5) coincides with the solution of 
(4) and with the original sparse vector x. Figure 3 provides an intuitive explanation 
why ¢;-minimization promotes sparse solutions. Here, N = 2 and m = 1, so one 


Fig. 3 The £,-minimizer 
within the affine space of 
solutions of the linear system 
Az = y coincides with a 
sparsest solution 
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deals with a line of solutions F(y) = {z : Az = y} in R?. Except for pathological 
situations where ker A is parallel to one of the faces of the polytope B?, there is a 
unique solution of the €;-minimization problem, which has minimal sparsity, i.e., 
only one nonzero entry. 

Recovery results in the next sections make rigorous the intuition that ¢,- 
minimization indeed promotes sparsity. 

For sparse recovery via greedy algorithms, one refers to the literature [151, 153]. 


The Null Space Property 
The null space property is fundamental in the analysis of £;-minimization. 


Definition 1. A matrix A € C’*" is said to satisfy the null space property (NSP) 
of order k with constant y € (0, 1) if 


lari S yllnrelh. 
for all sets T C {1,..., N}, #7 <k and for all 7 € ker A. 
The following sparse recovery result is based on this notion. 


Theorem 1. Let A ¢ C”*™ be a matrix that satisfies the NSP of order k with 
constant y € (0,1). Let x € C% and y = Ax and let x* be a solution of the 
£,-minimization problem (5). Then 


| 2(1+ y) 


|x —x" |i < ay (6) 


In particular, if x is k-sparse, then x* = x. 
Proof. Let n = x* — x. Then n € ker A and 
IIx" lh < Il 


because x* is a solution of the £,-minimization problem (5). Let T be the set of the 
k-largest entries of x in absolute value. One has 


llerlly + Wapelly S ler + [ere lls. 
It follows immediately from the triangle inequality that 
llr lla = Wr tla + Ure ll = Ieere lly S leer lh + [here lh. 
Hence, 


IInrelly S Marla + 2lkerell, S yilarella + 20%), 
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or, equivalently, 


Inre lla < OK (X)1. (7) 


Loy 


Finally, 


lx — x" = Marl + Mazel SY + Dllnrells S OK(X)1 


2(1+ y) 
l= 
and the proof is completed. a 


One can also show that if all k-sparse x can be recovered from y = Ax using 
£,-minimization, then necessarily A satisfies the NSP of order k with some constant 
y € (0, 1) [38,87]. Therefore, the NSP is actually equivalent to sparse £)-recovery. 


The Restricted Ilsometry Property 


The NSP is somewhat difficult to show directly. The restricted isometry property 
(RIP) is easier to handle, and it also implies stability under noise as stated below. 


Definition 2. The restricted isometry constant 5, of a matrix A € C”*” is the 
smallest number such that 


(1 — &)iizlls < Azlls < + be )IlzI5. (8) 
for all z © Xx. 


A matrix A is said to satisfy the restricted isometry property of order k with constant 
bx if d¢ € (0, 1). It is easily seen that 5, can be equivalently defined as 


bn = max || AF Ar — Id ||2+2, 
TCU,..N PAT Sk 


which means that all column submatrices of A with at most k columns are required 
to be well conditioned. The RIP implies the NSP as shown in the following lemma. 


Lemma 2. Assume that A € C”™ satisfies the RIP of order K = k +h with 
constant 5x € (0,1). Then A has the NSP of order k with constant y = ,/ k rok 


Proof. Letn € N = kerA andT Cc {I,...,N}, #7 < k. Define T) = T 
and 7\, T>,..., 7, to be disjoint sets of indexes of size at most h, associated to a 
nonincreasing rearrangement of the entries of 7 € N, i.e., 


Inj} <\|ni| forall j € Tie Ty, €>l' > 1. (9) 
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Note that An = 0 implies Annur, = — par =) Anr;. Then, from the Cauchy— 
Schwarz inequality, the RIP, and the triangle inequality, the following sequence of 


inequalities is deduced: 


llarlh < Vkllorll2 < Vkllnmun lle 


Ee any er ee ee I 
= 1_ Be NTU = 1_ be NTMH)UT3U+-UT, 2 


k - 1+éx s 
1 dy! nr; | Tae VED nl (10) 


It follows from (9) that || < |ne| for alli € T;41 and é € T;. Taking the sum over 
¢ € T; first and then the €2-norm overi € T; +1 yields 


-1/2 


Im <A aryl, and lrg lla SAT |r; Ih. 


Using the latter estimates in (10) gives 


1+ - 
larlh < Im lis fa = * In llr. (11) 


and the proof is finished. a 


Taking h = 2k above shows that 63, < 1/3 implies y < 1. By Theorem 1, 
recovery of all k-sparse vectors by £,-minimization is then guaranteed. Addition- 
ally, stability in £; is also ensured. The next theorem shows that RIP implies also a 
bound on the reconstruction error in £5. 


Theorem 2. Assume A € C”*% satisfies the RIP of order 3k with 63, < 1/3. For 
x € CN, let y = Ax and x* be the solution of the £,-minimization problem (5). 
Then 


OK(X)1 


Vk 


IIx -x"ll2<€ 


; _ 2 (yt _ [Fb 
with C = 75, (Ge + v), Y= Va): 


Proof. Similarly as in the proof of Lemma 2, denote 7 = x* — x € N = ker A, 
To = T the set of the 2k-largest entries of 7 in absolute value, and T; s of size 
at most & corresponding to the nonincreasing rearrangement of 7. Then, using (10) 
and (11) with h = 2k of the previous proof, 
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1+ d3, 12 
< c 
lela = aq ape lars. 


From the assumption 63, < 1/3, it follows that y := 
and 2 yield 


1+53, 


Tia) ~ 1. Lemmas 1 


llnrell2 = ox (m)2 < (2K) Minh = 2K“? (Une lh + Une ll) 


- y+ ue —1/2 
< (2k)? (yilnrella + Mle lly) S$ Sk llnzelh. 
Va * 
Since T is the set of 2k-largest entries of 7 in absolute value, it holds 


Ine |l1 = Il 7(supp xjox)° Il = Il (supp xyq)° Ih (12) 


where xj] is the best k-term approximation to x. The use of this latter estimate, 
combined with inequality (7), finally gives 


* +1 
Ix — x" lo < Ilnrlla + Ilnrella S (S * y)k nee lh 
2 y+ -4 
< —— {| ——| ae : 
= y ( 2 Tr r) OK(x)1 
This concludes the proof. a 


The restricted isometry property implies also robustness under noise on the 
measurements. This fact was first noted in [26, 30]. 


Theorem 3. Assume that the restricted isometry constant 52% of the matrix A € 
C”* satisfies 


bo < 1/2 = 0.7071 (13) 


Then the following holds for all x € C. Let noisy measurements y = Ax + e be 
given with |e||2 <n. Let x* be the solution of 


min ||z||; subject to || Az — y|]2 < 7. (14) 
Then 


OK(X)1 


Vk 


for some constants C,, Cy > 0 that depend only on 8x. 


Ix — x" |l2 < Cin + Co——— 
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The constant in (13) was improved several times [22, 38, 74-76] until the present 
statement was reached in [19], which is actually optimal [45]. 


Coherence 


The coherence is a by now classical way of analyzing the recovery abilities of a 
measurement matrix [50, 151]. For a matrix A = (a;|a2|---|ay) € C”*® with 
normalized columns, ||a¢||2 = 1, it is defined as 


:= max |(a¢, ax)|. 
ye = max (ae. ax) 


Applying Gershgorin’s disc theorem [93] to AF, Ar — I with #7 = k shows that 
bk < (k—I)p. (15) 


Several explicit examples of matrices are known which have small coherence 
pu = O(1/,/m). A simple one is the concatenation A = (J|F) € C”*?” of the 
identity matrix and the unitary Fourier matrix F ¢€ C”*” with entries Fj, = 
m—'/2e2rtik/m Tt is easily seen that 1. = 1/./m in this case. Furthermore, [143] 
gives several matrices A € cmxm* with coherence pL = 1/./m. In all these cases, 
oR <C aa Combining this estimate with the recovery results for €;-minimization 


above shows that all k-sparse vectors x can be (stably) recovered from y = Ax via 
£,-minimization provided 


m > C'k?. (16) 


At first sight, one might be satisfied with this condition since if k is very small 
compared to N, then still m might be chosen smaller than N and all k-sparse vectors 
can be recovered from the undersampled measurements y = Ax. Although this is 
great news for a start, one might nevertheless hope that (16) can be improved. In 
particular, one may expect that actually a linear scaling of m in k should be enough 
to guarantee sparse recovery by £,-minimization. The existence of matrices, which 
indeed provide recovery conditions of the form m > Ck log*(N) (or similar) with 
some a > 1, is shown in the next section. Unfortunately, such results cannot be 
shown using simply the coherence because of the generally lower bound [143] 


oy ea : (N sufficiently large) 
~ suiticien arge). 
il m(N—-1)  /m 7 


In particular, it is not possible to overcome the “quadratic bottleneck” in (16) by 
using Gershgorin’s theorem or Riesz-Thorin interpolation between ||- ||;1 and 
Il - lloo-+003 See also [131, 141]. In order to improve on (16), one has to take into 
account also cancellations in the Gramian Aj. Ar — J, and this task seems to be quite 
difficult using deterministic methods. Therefore, it will not come as a surprise that 
the major breakthrough in compressive sensing was obtained with random matrices. 
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It is indeed easier to deal with cancellations in the Gramian using probabilistic 
techniques. 


RIP for Gaussian and Bernoulli Random Matrices 


Optimal estimates for the RIP constants in terms of the number m of measurement 
matrices can be obtained for Gaussian, Bernoulli, or more general subgaussian 
random matrices. 

Let X be a random variable. Then one defines a random matrix A = A(@), w € 
(2, as the matrix whose entries are independent realizations of X, where (Q, X, P) 
is their common probability space. One assumes further that for any x € R™ one 
has the identity E|| Ax||3 = ||x||5, E denoting expectation. 

The starting point for the simple approach in [7] is a concentration inequality of 
the form 


P (||/Axll3 — xll3] = Six) < 2e7? ™,  O0< 8 <1, (17) 


where cg > 0 is some constant. 
The two most relevant examples of random matrices which satisfy the above 
concentration are the following: 


1. Gaussian matrices. Here the entries of A are chosen as 1.i.d. Gaussian random 
variables with expectation 0 and variance 1/m. As shown in [1], Gaussian 
matrices satisfy (17). 

2. Bernoulli matrices The entries of a Bernoulli matrices are independent realiza- 
tions of +1/./m Bernoulli random variables, that is, each entry takes the value 
+1/,./m or —1/./m with equal probability. Bernoulli matrices also satisfy the 
concentration inequality (17) [1]. 


Based on the concentration inequality (17), the following estimate on RIP 
constants can be shown [7, 26, 76, 110]. 


Theorem 4. Assume A € R”*" to be a random matrix satisfying the concentration 
property (17). Then there exists a constant C depending only on co such that the 
restricted isometry constant of A satisfies 5, < 6 with probability exceeding | — € 
provided 


m > C8*(k log(N/m) + log(e7')). 
Combining this RIP estimate with the recovery results for €;-minimization shows 
that all k-sparse vectors x € C% can be stably recovered from a random draw of A 


satisfying (17) with high probability provided 


m > Cklog(N/m). (18) 
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Up to the logarithmic factor, this provides the desired linear scaling of the number 
m of measurements with respect to the sparsity k. Furthermore, as shown in Sect. 3 
below, condition (18) cannot be further improved; in particular, the log-factor cannot 
be removed. 

It is useful to observe that the concentration inequality is invariant under unitary 
transforms. Indeed, suppose that z is not sparse with respect to the canonical basis 
but with respect to a different orthonormal basis. Then z = Ux for a sparse x anda 
unitary matrix U ¢ C’*", Applying the measurement matrix A yields 


Az = AUx, 


so that this situation is equivalent to working with the new measurement matrix 
A’ = AU and again sparsity with respect to the canonical basis. The crucial point 
is that A’ satisfies again the concentration inequality (17) once A does. Indeed, 
choosing x = U~'x’ and using unitarity gives 


P (AU — |ll3| = 8ilxll2y) = P (|Ax'IB _yu-x'Ip] = s1U-1s'h) 
=P (|Ax’ [3 — x’ IB] > Blix’ Py) < 22. 


Hence, Theorem 4 also applies to A’ = AU. This fact is sometimes referred to as 
the universality of the Gaussian or Bernoulli random matrices. It does not matter 
in which basis the signal x is actually sparse. At the coding stage, where one takes 
random measurements y = Az, knowledge of this basis is not even required. Only 
the decoding procedure needs to know U. 


Random Partial Fourier Matrices 


While Gaussian and Bernoulli matrices provide optimal conditions for the minimal 
number of required samples for sparse recovery, they are of somewhat limited use 
for practical applications for several reasons. Often the application imposes physical 
or other constraints on the measurement matrix, so that assuming A to be Gaussian 
may not be justifiable in practice. One usually has only limited freedom to inject 
randomness in the measurements. Furthermore, Gaussian or Bernoulli matrices are 
not structured, so there is no fast matrix-vector multiplication available which may 
speed up recovery algorithms, such as the ones described in Sect. 4. Thus, Gaussian 
random matrices are not applicable in large-scale problems. 

A very important class of structured random matrices that overcomes these 
drawbacks are random partial Fourier matrices, which were also the object of study 
in the very first papers on compressive sensing [26, 29, 128, 129]. A random partial 
Fourier matrix A €¢ C”** is derived from the discrete Fourier matrix F ¢ C’*% 
with entries 
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by selecting m rows uniformly at random among all N rows. Taking measurements 
of a sparse x € C% corresponds then to observing m of the entries of its 
discrete Fourier transform X = Fx. It is important to note that the fast Fourier 
transform may be used to compute matrix-vector multiplications with A and A* 
with complexity O(N log(V)). The following theorem concerning the RIP constant 
was proven in [131] and improves slightly on the results in [26, 129, 138]. 


Theorem 5. Let A € C”*% be the random partial Fourier matrix as just described. 
Then the restricted isometry constant of the rescaled matrix ,| XA satisfies 6, <6 


with probability at least 1 — NY los?) provided 
m > C8 7k log'(N). (19) 


The constants C, y > 1 are universal. 


Combining this estimate with the £;-minimization results above shows that recovery 
with high probability can be ensured for all k-sparse x provided 


m > Cklog*(N). 


The plots in Fig. | illustrate an example of successful recovery from partial Fourier 
measurements. 

The proof of the above theorem is not straightforward and involves Dudley’s 
inequality as a main tool [131, 138]. Compared to the recovery condition (18) 
for Gaussian matrices, one suffers a higher exponent at the log-factor, but the 
linear scaling of m in k is preserved. Also a nonuniform recovery result for £- 
minimization is available [29, 128, 131], which states that each k-sparse x can 
be recovered using a random draw of the random partial Fourier matrix A with 
probability at least 1—e provided m > Ck log(N/e). The difference to the statement 
in Theorem 5 is that for each sparse x, recovery is ensured with high probability for 
a new random draw of A. It does not imply the existence of a matrix which allows 
recovery of all k-sparse x simultaneously. The proof of such recovery results does 
not make use of the restricted isometry property or the null space property. 

One may generalize the above results to a much broader class of structured 
random matrices which arise from random sampling in bounded orthonormal 
systems. The interested reader is referred to [128, 129, 131, 132]. 

Another class of structured random matrices, for which recovery results are 
known, consist of partial random circulant and Toeplitz matrices. These correspond 
to subsampling the convolution of x with a random vector b at m fixed (determin- 
istic) entries. The reader is referred to [130, 131, 133] for detailed information. 
Near-optimal estimates of the RIP constants of such type of random matrices 
have been established in [101]. Further types of random measurement matrices are 
discussed in [101, 122, 124, 137, 154]; see also [99] for an overview. 
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Compressive Sensing and Gelfand Widths 


In this section a quite general viewpoint is taken. The question is investigated how 
well any measurement matrix and any reconstruction method — in this context 
usually called the decoder — may perform. This leads to the study of Gelfand 
widths, already mentioned in Sect. 2. The corresponding analysis allows to draw 
the conclusion that Gaussian random matrices in connection with £,-minimization 
provide optimal performance guarantees. 

Following the tradition of the literature in this context, only the real-valued case 
will be treated. The complex-valued case is easily deduced from the real-valued case 
by identifying C’ with R*” and by corresponding norm equivalences of £ p-horms. 

The measurement matrix A ¢€ R”* is here also referred to as the encoder. 
The set A,, 7 denotes all possible encoder/decoder pairs (A, A) where A € RN 
and A: R” > RY is any (nonlinear) function. Then, for 1 < k < N, the 
reconstruction errors over subsets K C R%, where R% is endowed with a norm 
|| - lly, are defined as 


ox (K)x := sup ox (x)x, 
xe€K 


II 


En(K,X): inf — sup ||x — A(Ax)|lx. 
K 


(A,A)EAmN xE 


In words, E,,(K,X) is the worst reconstruction error for the best pair of 
encoder/decoder. The goal is to find the largest & such that 


En(K, xX) < Coo, (K) x. 


Of particular interest for compressive sensing are the unit balls K = B a for 0 < 


p<iandX = a because the elements of B are well approximated by sparse 
vectors due to Lemma |. The proper estimate of E,,(K, X) turns out to be linked to 
the geometrical concept of Gelfand width. 


Definition 3. Let K be a compact set in a normed space X. Then the Gelfand width 
of K of order m is 


d™(K, X):= inf sup{||xllx :x Ee KN Y}, 
YCX 


codim(Y) < m 


where the infimum is over all linear subspaces Y of X of codimension less or equal 
tom. 


The following fundamental relationship between F,,(K, X) and the Gelfand 
widths holds. 
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Proposition 1. Let K C R% be a closed compact set such that K = —K and 
K+ K C CoK for some constant Co. Let X = (IR, ||-||x) be a normed space. 
Then 


d™(K, X) < E,(K, X) < Cod" (K, X). 


Proof. For a matrix A € R”*, the subspace Y = ker A has codimension less or 
equal to m. Conversely, to any subspace Y C R% of codimension less or equal to 
m, a matrix A € R”*% can be associated, the rows of which form a basis for Y~. 
This identification yields 


d™(K,X)= inf sup{|lylly : 7 ekerAN K}. 
AERMXN 


Let (A, A) be an encoder/decoder pair in A», and z = A(0). Denote Y = ker(A). 
Then with 7 € Y also—n € Y, and either ||7 —z||x > ||n||x or || —7—Zllx = |lnllx- 
Indeed, if both inequalities were false, then 


ll2nllx = Iln-—z+z+nllx S IIn—zllx + ll-n—allx < 2ilnllx, 


a contradiction. Since K = —K, it follows that 


d"(K, X) inf sup{|lnlly:n€YOK}< sup |ln—2zllx 
AERMXN 


nEeYNK 


sup ||7 — A(An)|lx < sup ||x — A(Ax)|lx. 
neYNK xE€K 


Taking the infimum over all (A, A) € Amy yields 
d"(K,X) < En(K,X). 
To prove the converse inequality, choose an optimal Y such that 
d™(K, X) = sup{||x|ly : x € YN K}. 


(An optimal subspace Y always exists [107].) Let A be a matrix whose rows form a 
basis for Y+. Denote the affine solution space F(y) := {x : Ax = y}. One defines 
then a decoder as follows. If F(y) N K # @, then choose some x(y) € F(y) and 
set A(y) = x(y). If F(y) N K = @, then A(y) € F(y). The following chain of 
inequalities is then deduced: 


Em(K,X)< sup — sup |x — x’ Lx 
yY x,x/EF(y)NK 


< sup |nilx < Cod™(K, X), 


n€Co(YNK) 


which concludes the proof. a 
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The assumption K + K C CoK clearly holds for norm balls with Co = 2 and 
for quasi-norm balls with some Cp > 2. The next theorem provides a two-sided 
estimate of the Gelfand widths a(Be vy) [48, 77, 157]. Note that the case p = 1 
was considered much earlier in [77, 79, 96]. 


Theorem 6. Let 0 < p < 1. There exist universal constants C,, Dp > 0 such that 
the Gelfand widths d™ (BN , £') satisfy 


1/p—1/2 
Cp min} 1, “PPO < d"(BY, £7) 


m 


(20) 


In(2N a 
< Dy min 1, — 


m 


Combining Proposition | and Theorem 6 gives in particular, for large m, 


Gy SEEN) < E,(BY 08) < py See (21) 


This estimate implies a lower estimate for the minimal number of required samples 
which allows for approximate sparse recovery using any measurement matrix and 
any recovery method whatsoever. The reader should compare the next statement 
with Theorem 2. 


Corollary 1. Suppose that A € R”*" and A : RR” — RN such that 


Ok (x)1 


IIx — A(AX)I2 = C Jk 


forall x € Br and some constant C > 0. Then necessarily 
m > C'k log(2N/m). (22) 


Proof. Since ox(x)1 < ||x||1 < 1, the assumption implies E,n(BN , €) < Ck7'?, 
The lower bound in (21) combined with Proposition | yields 


~ flog(2 
Gi.) SE < EAR) £Ck, 
m 


Consequently, m > C’k log(eN/m) as claimed. (3 


In particular, the above lemma applies to £;-minimization and consequently 6, < 
0.5 (say) for a matrix A € R”*% implies m > Cklog(N/m). Therefore, the 
recovery results for Gaussian or Bernoulli random matrices with £)-minimization 
stated above are optimal. 
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It can also be shown that a stability estimate in the £;-norm of the form ||x — 
A(Ax)||1 < Cox(x); for all x € R implies (22) as well [46,77]. 


Extensions of Compressive Sensing 


The problem of recovering nearly sparse vectors from a minimal number of 
nonadaptive linear measurements can be extended in several forms. In the following 
two possible, mutually related, extensions are addressed. 

The first refers to the object to be recovered: while in the theory of compressive 
sensing presented so far nearly sparse vectors are the unknowns of the problem 
at hand, one may consider as well matrices of approximate low rank. This leads 
to the problem of finding a matrix of minimal rank consistent with a given 
underdetermined linear system of equations. 

The second generalization of compressive sensing considers nonlinear nonadap- 
tive measurements. The simplest nonlinearity consists in quadratic measurements, 
given by the squared magnitude of the scalar products of the vector with respect to a 
fixed basis or a frame. The associated recovery task — called the phase retrieval 
problem — appears in many physical situations where only intensity values can 
be observed. More generally, one may consider the recovery from higher-order 
measurements. Surprisingly enough, polynomial-type, and in particular quadratic, 
measurements of vectors can be recast into an affine low-rank minimization problem 
discussed above, creating a connection between these two extensions of compressive 
sensing. 


Affine Low-Rank Minimization 

Here one denotes the space of real or complex n x p matrices by M,x,. Given a 
linear map .Y : Myxp — C”, with m < pn, and a vector y € C’”, one considers 
the affine rank minimization problem 


min rank(X) subject to .“(X) = y. (23) 
E 


nXp 


An important special case of low-rank matrix recovery is matrix completion, where 
/ samples a few entries, 


S(X)e = xij, (24) 


for some i, j depending on £. 

Like for the £9-minimization problem (4), the affine rank minimization problem 
is NP-hard in general [76, 114, 135]; therefore, it is desirable to have tractable 
alternatives. In [67], Fazel studied nuclear norm minimization for this purpose. 
Here, the nuclear norm || X||« of a matrix X is the ¢,-norm of its singular values 
and is the largest convex envelope of the rank function [67]. One solves then 


min |X|]. subject to. “(X) = y. (25) 
xem, 


nxXp 
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There are two known regimes where nuclear norm minimization can be guaran- 
teed to return minimal-rank solutions: 


RIP measurement maps 
The (rank) restricted isometry property is analogous to the classical restricted 
isometry property from compressive sensing already mentioned in Sect. 3: 


Definition 4 (Rank-Restricted Isometry Property [135]). Let 7 : Mix, > C” 
be a linear map. For an integer k, with 1 < k <n, define the k-restricted isometry 
constant 6; = 6,(.%) > 0 to be the smallest number such that 


(1 = 8) IX Me SA OO ip S + 8) XU 
holds for all k-rank matrices X. 


It is known that nuclear norm minimization (25) recovers all matrices X of rank 
at most k from the measurements y = .7(X), provided 62, < 1/ /2; see [19, 24, 
76, 121, 135]. 

The restricted isometry property is known to hold for Gaussian (or more 
generally subgaussian) measurement maps [24, 135], which are of the form 


S(X)e = Yo aenj Xe.j, €=1,...,m, (26) 
kj 


where the a¢,,; are independent normal distributed random variables with mean 
zero and variance 1/m. Such a map satisfies 6, < 6 € (0, 1) with high probability 
provided 


m => Cs max{ p,n}k; (27) 


see Theorem 2.3 in [24]. Since the degrees of freedom of a rank k matrix X € Mix) 
are k(n + p —k), the above bound matches this number up to possibly a constant. 
Therefore, the bound (27) is optimal. 

It follows from recent results in [3, 100] that the restricted isometry property 
also holds for certain structured random maps if slightly more measurements are 
allowed. In particular, let 7% = Po FD, where D : Mixy — Mhix performs 
independent random sign flips of all entries of a matrix, F : Myx, > Mhxp 
represents the (suitably normalized) 2D Fourier transform, and Pg : Maxp > C” 
is the coordinate projection which extracts the entries of the matrix on a random set 
Q C [n] x [p] which is chosen uniformly at random among all subsets of cardinality 
m. Then the rank-restricted isometry constants of .Y satisfy 6, < 6 € (0,1) with 
high probability as long as 


m > Cs max{ p,n}k log? (pn). (28) 
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This follows from recent findings in [3, 100] for which such random partial Fourier 
measurements satisfy a concentration inequality of the form 


m = 
P( (IOI? — IX1F| 2 eX) <2exp-TC.log“(pn)), 29) 


for e € (0,1), together with the same proof strategy employed for subgaussian 
measurement maps based on covering arguments [24,76]. This was first noted in 
the introduction of [73]. 


Matrix Completion 

In the matrix completion setup (24) where measurements are pointwise observations 
of entries of the matrix, there are obvious rank one matrices in the kernel of the 
operator .7; therefore, the RIP fails completely in this setting, and “localized” low- 
rank matrices in the null space of .7 cannot be recovered by any method whatsoever. 
However, if certain conditions on the left and right singular vectors of the underlying 
low-rank matrix are imposed, essentially requiring that such vectors are uncorrelated 
with the canonical basis, then it was shown in [25, 27, 134] that such incoherent 
matrices of rank at most k can be recovered from m randomly chosen entries with 
high probability provided 


m > Ckmax{n, p} log”(max{n, p}). 


Up to perhaps the exponent 2 at the log-term, this is optimal. One refers to [25, 27, 
134] for detailed statements. In [88,89] extensions to quantum state tomography can 
be found. 


Nonlinear Measurements 


Phase Retrieval 

In order to understand how one can generalize compressive sensing towards nonlin- 
ear measurements, it is instructive to start with the phase retrieval problem. Here, 
the measurements of a signal x are given by y; = |(a;,x)|? for some vectors a;, 
i = 1,...,m_and the task is to reconstruct x up to a global phase factor. Practically 
successful approaches can be found in the physics and optimization literature for 
well over a decade; see, for instance, [8, 68]. In these investigations, however, 
neither bounds of the minimal number nor a theoretical discussion on the type of 
measurements is provided for the phase retrieval problem to ensure reconstruction. 
Moreover, there are no guarantees for these algorithms to converge to the expected 
solution. Only recently, uniqueness of feasible solutions was provided by Balan 
et al. [5] using methods from algebraic geometry. The authors proved injectivity for 
signals x € R‘ and m > 2N — 1 generic measurements or for complex signals 
x € C% andm > 4N — 2 generic measurements. Unfortunately, this theoretical 
work did not provide recovery guarantees for a practical algorithm. The main tool 
in this work was the observation that quadratic measurements can be written as 
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vi = lai, x)? = (aa? xx"), 2 = 10.25, (30) 


where the inner product on the right-hand side here denotes the Hilbert-Schmidt 
inner product (A, B) = Tr(AB*). Consequently x can be identified as the unique 
solution of the phase retrieval problem (up to a global phase factor) if X = xx* is 
the unique rank one (and thus the minimum rank!) positive semidefinite solution of 
the linear equations 


yi = S(X); = (aia; X), i=1,...,m. (31) 


This well-known lifting trick is also the basis of the PhaseLift algorithm by Candés 
et al. [32]. Here, the main ingredient is to pass from the rank minimization problem 
constrained by (31) to its convex relaxation (25). Using techniques from random 
matrix theory, the authors showed recovery guarantees form = O(N log(N)) 
Gaussian measurements under no structural (sparsity) assumptions on the signal. 
Although PhaseLift is often not considered a very efficient approach (see Sect. 4 for 
possible implementations) as one needs to solve a semidefinite program for N x N 
matrices — thus squaring the dimension — this work is considered groundbreaking, as 
for the first time it provided recovery guarantees for a polynomial time algorithm. 
The number of Gaussian measurements for PhaseLift to recover a generic signal 
has later been improved to m = O(N) in [23] and for a variant of PhaseLift to 
m = O(k? log(N)) for k-sparse signals in [105]. These recovery guarantees include 
also stability with respect to measurement noise. 

The question of how many measurements are necessary to allow for stable phase 
retrieval, independently of the algorithm used, has been addressed in a very general 
setting by Eldar and Mendelson [64]. They showed that the necessary number of 
phaseless measurements to allow for the recovery of a signal x which is known to 
lie in a given set S can be estimated in terms of the so-called Gaussian width of 
the set S. This result is not restricted to Gaussian measurements but extends also to 
measurement vectors with independent subgaussian entries which in addition satisfy 
a small ball probability bound. Building upon this work, Ehler et al. [60] showed 
recovery guarantees for certain greedy algorithms for complex measurements of the 
same type. Other algorithms, for which recovery guarantees have been provided for 
Gaussian measurements, include polarization-based approaches [4] and alternating 
minimization [116]. 

Rigorous recovery guarantees for phase retrieval have been extended to randomly 
masked Fourier transforms in [31,90] and to spherical designs in [91]. These works 
address more realistic measurement scenarios than Gaussian measurements for 
practical applications such as X-ray crystallography; see also [99]. 


Higher-Order Polynomial Measurements 
Building upon the successful experience on phase retrieval problems, compressed 


sensing theory can be further extended to solving nonlinear problems of the type 


min||z|lo subject to y; = Aj(z), i =1,...,m, (32) 
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where A; : CY — C,i = 1,...,m, are smooth functions. For g being a sufficiently 
large even integer, one approximates A; by its gth-order Taylor expansion, 


Ai(x)* >> om DEA eA Gayest Ae, 


05 |a|<q 


: N+4 N+¢4 : : ad 

where A; is a ( A 2 )x i 2 )-symmetric matrix and x is the vector whose 
2. 2. 

entries are all the monomials of the elements of x with degree less or equal to g/2. 

Hereby, the standard multi-index notation is used. 


Example 1. Let x = (x1,X2)' and A(x) = 14+ x1 + xe. The 4th-order Taylor 
expansion around x9 = 0 is given by 


A(x) = x* Ax 
where x = (Lt aap ity HF x)? and 
1 1/200 0 0 


1/2 0 001/120 
0 0 00 0 0 


A=! 9 0 000 0 
0 1/1200 0 O 
0 0 00 0 0 
One observes again that 
y= x* Ax = (Aj, ¥x*), i=1,...,m, (33) 


where the inner product on the right-hand side denotes the Hilbert-Schmidt inner 
product. Consequently x can be recognized as the solution of (32) if X = xx* is 
the unique rank one positive definite solution of the linear equations 


= (pa (A Kh PS 1 cath: (34) 


As for the PhaseLift algorithm for phase retrieval problems, one considers again 
instead of the rank minimization problem constrained by (34) its convex relaxation 
(25). One also notices that in case the vector x is sparse, then X is sparse in addition 
to being of rank one. Hence, one can combine the nuclear norm with the classical 
£,-norm penalization and consider the problem 


Peer |X lx + y [|X lle, subject to .7(X) = y, (35) 
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q 
where p = (“ : ’) and y > 0 is a suitable parameter. One refers to (35) 


2 
as nonlinear basis pursuit. This algorithm works surprisingly well in practice 
to recover sparse vectors from very few measurements [118], but theoretical 
guarantees for stable and unique recovery are open to date. 


Applications 


Compressive sensing can be potentially used in all applications where the task is 
the reconstruction of a signal or an image from linear measurements, while taking 
many of those measurements — in particular, a complete set of measurements — is a 
costly, lengthy, difficult, dangerous, impossible, or otherwise undesired procedure. 
Additionally, there should be reasons to believe that the signal is sparse in a suitable 
basis (or frame). Empirically, the latter applies to most types of signals. 

In computerized tomography, for instance, one would like to obtain an image of 
the inside of a human body by taking X-ray images from different angles. Taking 
an almost complete set of images would expose the patient to a large and dangerous 
dose of radiation, so the amount of measurements should be as small as possible 
and nevertheless guarantee a good enough image quality. Such images are usually 
nearly piecewise constant and therefore nearly sparse in the gradient, so there is a 
good reason to believe that compressive sensing is well applicable. And indeed, it 
is precisely this application that started the investigations on compressive sensing in 
the seminal paper [29]. 

Also radar imaging seems to be a very promising application of compressive 
sensing techniques [65, 66,94, 144]. One is usually monitoring only a small number 
of targets, so that sparsity is a very realistic assumption. Standard methods for radar 
imaging actually also use the sparsity assumption, but only at the very end of the 
signal processing procedure in order to clean up the noise in the resulting image. 
Using sparsity systematically from the very beginning by exploiting compressive 
sensing methods is therefore a natural approach. 

Further potential applications include wireless communication [147], astronom- 
ical signal and image processing [14], analog to digital conversion [154], camera 
design [58], and imaging [136, 159, 160]. 

Affine low-rank minimization problems (23) arise in many areas of science and 
technology, including system identification [113], collaborative filtering, quantum 
state tomography [88, 89], signal, and image processing. An important special case 
is the matrix completion problem [25, 27, 134], where the task consists in filling 
in missing entries of a large low-rank data matrix. For applications of low-rank 
matrix recovery in quantum state tomography, one can refer to [88,89]. Applications 
of phase retrieval include diffraction imaging and X-ray crystallography; see, for 
instance, [111]. 
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4 Numerical Methods 


The previous sections showed that £,-minimization performs very well in recovering 
sparse or approximately sparse vectors from undersampled measurements. In appli- 
cations, it is important to have fast methods for actually solving €,-minimization 
problems. Two such methods — Chambolle and Pock’s primal-dual algorithm [34] 
and iteratively re-weighted least squares (IRLS) [44] — will be explained in more 
detail below. 

As a first remark, the €;-minimization problem 


min ||x||1 subject to Ax = y (36) 


is in the real case equivalent to the linear program 


2N 
min ) > v;  subjectto v>0, (A|—A)v=y. (37) 


j=l 


The solution x* to (36) is obtained from the solution v* of (37) via x* = 
(Id | — Id)v*. Any linear programming method may therefore be used for solving 
(36). The simplex method and interior point methods apply in particular [115], 
and standard software may be used. (In the complex case, (36) is equivalent to 
a second-order cone program (SOCP) and can also be solved with interior point 
methods.) However, such methods and software are of general purpose, and one may 
expect that methods specialized to (36) outperform such existing standard methods. 
Moreover, standard software often has the drawback that one has to provide the 
full matrix rather than fast routines for matrix-vector multiplication which are 
available, for instance, in the case of partial Fourier matrices. In order to obtain 
the full performance of such methods, one would therefore need to reimplement 
them, which is a daunting task because interior point methods usually require 
much fine-tuning. On the contrary, the two specialized methods described below 
are rather simple to implement and very efficient. Many more methods are available 
nowadays, including the homotopy and LARS method [55, 59, 119, 120]; greedy 
methods; such as orthogonal matching pursuit [151], CoSaMP [153]; and iterative 
hard thresholding [13,69], which may offer better complexity than standard interior 
point methods. Due to space limitations, however, only the two methods below are 
explained in detail. 


A Primal-Dual Algorithm 


The reconstruction approaches discussed in the previous section can often be solved 
by optimization problems of the form 
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min F(Ax) + G(x) (38) 
xeCr 


with A being an m x N matrix and F,G being convex functions with values in 
(—oo, oo]. For instance, the £;-minimization problem (5) can be recast as (38) with 
G(x) = |[xll1 and F(z) = y4;(2), where y,,} is the characteristic function of 
the singleton set {y} that takes the value 0 on y and oo elsewhere. Moreover, (14) 
is equivalent to (38) with G(x) = ||x||; and F = Y¢y:1y—-w2<7}- Also the nuclear 
norm minimization problem (25) takes the form (38) with G being the nuclear norm; 
see also below. 
The so-called dual problem [17,76] of (38) is given by 


max — F*(&) — G*(—A*8), (39) 
gecn 


where 
F*(E) = sup{Re(z, &) — F(@):z¢C™} 


is the convex (Fenchel) conjugate of F and likewise G* is the convex conjugate 
of G. The joint primal-dual optimization is equivalent to solving the saddle point 
problem 


min max Re(Ax, &) + G(x) — F*(€). (40) 
xeCN &eC™ 


The primal-dual algorithm introduced by Chambolle and Pock in [34], generalizing 
[126], and described below solves this problem iteratively. In order to formulate it, 
one needs to introduce the so-called proximal mapping (proximity operator). For 
the convex function G, it is defined as 


1 
Pg(t,z) := argmin tG(x) + =||x —2([3 
xeCN 2 


For G(x) = ||x|l1, the proximal mapping reduces to the well-known soft- 
thresholding operator, which is defined componentwise as 


sgn(ze)(|ze| — t) if |ze| = t, 
S, a 
We 0 otherwise. 


One also requires below the proximal mapping associated to the convex conjugate 
y kl 

The primal-dual algorithm performs an iteration on the dual variable, the primal 
variable, and an auxiliary primal variable. Starting from initial points x°, ¥° = x € 
C% ,&° € C” and parameters t,o > 0, 6 € [0, 1], one iteratively computes 
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E"tl = Ppx(o;&" + cAX"), (41) 
ntl = Pg(t; x" _ gate), (42) 
gti = ntl 4 O(x"t! — x"), (43) 


It can be shown that (x, &) is a fixed point of these iterations if and only if (x, &) 
is a saddle point of (40) which is equivalent to x being a solution of (38) and & of 
(39); see [76, Proposition 15.6]. For this reason, the algorithm can be interpreted as 
a fixed point iteration. 

For the case that 6 = 1, the convergence of the primal-dual algorithm has been 
established in [34]. 


Theorem 7. Consider the primal-dual algorithm (41)-(43) with 0 = 1 and t,o > 
0 such that to|| A||3_,, < 1. Ifthe problem (40) has a saddle point, then the sequence 
(x", &") converges to a saddle point of (40) and in particular (x") converges to a 
minimizer of (38). 


One observes that for the €; and nuclear norm minimization problems of this 
chapter, the assumption of the theorem that (40) has a saddle point will always be 
satisfied. An estimate of the convergence rate can be shown as well. One refers 
to [34] for details. Preconditioning was considered in [125], where also parameter 
choices are discussed. Convergence results for values of 6 different from 1 have 
been obtained in [92] for slight modifications of the algorithm. 

One considers again some special cases. For €;-minimization (36), where 
G(x) = |x|]; and F = y,,}, the proximal mapping Pg reduces to soft-thresholding 
(41) while the proximal mapping associated to F*(€) = Re(y, &) is given by 


Prx(o,€) = § —oy. 


Therefore, all operations required in the algorithm (41)-(43) are simple and easy 
to implement. Also, note that if fast matrix-vector multiplication algorithms are 
available for A and A*, these can be easily exploited. 

In the case of the nuclear norm minimization problem (25), where G(X) = 
|X |x and F = y,4,}, the proximal mapping of G is given by singular-value soft 
thresholding: If Z = UXV* is the singular value decomposition of a matrix Z 
with unitary matrices U and V and a diagonal matrix © = diag(o1,...,0,) with 
0, > 02>... > 0, then 


Py- |. Z) => U diag($7(01),--- »S2(On))V%, 


i.e., the soft-thresholding operator is applied to the singular values. With this 
information together with the proximal mapping of F* given above, the iterations 
(41)-(43) can be implemented for nuclear norm minimization. 
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Of course, the described algorithm applies to many more optimization problems 
and turns out to be very efficient when the proximal mappings are simple to evaluate. 
Notice that the iterations (41)-(43) are strongly influenced by backward—forward 
splitting methods [41] for the primal problem (38) with differentiable F’, which are 
of the form 


x” = Pg(t; x" — tVF(x")). (44) 


In fact, (41) can be considered a backward—forward step for the dual variable, 
while (42) is a backward—forward step for the primal variable. A particular instance 
of (44) is the soft-thresholding algorithm for minimizing functionals of the form 
|| Ax — yl]Z + Aljxlli; see [41, 43] and [9, 10] for an accelerated version. Related 
optimization algorithms include Douglas—Rachford splittings [39, 40,57] and the 
alternating direction method of multipliers [40, 83]. One refers to [142, Chapter 7], 
[40], [76, Chapter 15], and [156] for overviews on these and further approaches. 


Iteratively Re-weighted Least Squares 


This section is concerned with an iterative algorithm which, under the condition that 
A satisfies the NSP (see Definition 1), is guaranteed to reconstruct vectors with the 
same error estimate (6) as £;-minimization. The following discussion is restricted 
to the real-valued case. This algorithm has a guaranteed linear rate of convergence 
which can even be improved to a superlinear rate with a small modification. First, 
a brief introduction aims at shedding light on the basic principles of this algorithm 
and their interplay with sparse recovery and £,-minimization. 

Denote F(y) = {x : Ax = y} and .N = ker A. The starting point is the trivial 

2 


observation that |t| = nl for t # 0. Hence, an £,-minimization can be recasted into 


a weighted £-minimization, with the hope that 


N N 

: F 2) _*)-1 

arg min |x;| arg min xi\xjl, 
xEF(y) 4 1 xEF (y)“ 1 
i= i= 


as soon as x* is the desired £;-norm minimizer. The advantage of the reformulation 
consists in the fact that minimizing the smooth quadratic function f? is an easier 
task than the minimization of the nonsmooth function |t|. However, the obvious 
drawbacks are that neither one disposes of x* a priori (this is the vector one is 
interested to compute!) nor one can expect that x # O forall 7 = 1,..., N, since 
one hopes for k-sparse solutions. 

Suppose one has a good approximation w’ of [(xF)? $2)? w Een for 
some €, > 0. One computes 


x""" = arg min xiw" (45) 
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and then updates €,4; < €, by some rule to be specified later. Further, one sets 
1 1\2 2 |-1/2 
we tt = [tthy? + 2 (46) 


and iterates the process. The hope is that a proper choice of €, —> 0 allows the 
iterative computation of an £,-minimizer. The next sections investigate convergence 
of this algorithm and properties of the limit. 


Weighted £2-Minimization 
Suppose that the weight w is strictly positive which means that w; > 0 for all 
J €{il,...,N}. Then £2(w) is a Hilbert space with the inner product 


N 
(U,V) w = Dl wjujy;. (47) 
j=l 
Define 
x” := arg min ||zIlo», (48) 
zeF(y) 
where ||z||2,~ = (z, Z) 1.7. Because the || - |l2..-norm is strictly convex, the minimizer 


x” is necessarily unique; it is characterized by the orthogonality conditions 


(x",n)y=0, foralln EN. (49) 


An Iteratively Re-weighted Least Squares Algorithm (IRLS) 
An IRLS algorithm appears for the first time in the Ph.D. thesis of Lawson in 1961 


[102], in the form of an algorithm for solving uniform approximation problems. 
This iterative algorithm is now well known in classical approximation theory as 
Lawson’s algorithm. In [37] it is proved that it obeys a linear convergence rate. In 
the 1970s, extensions of Lawson’s algorithm for £,-minimization, and in particular 
€,-minimization, were introduced. In signal analysis, IRLS was proposed as a 
technique to build algorithms for sparse signal reconstruction in [86]. The interplay 
of the NSP, £)-minimization, and a re-weighted least square algorithm has been 
clarified only recently in the work [44]. 
The analysis of the algorithm (45) and (46) starts from the observation that 


: 1 2: —1 
t| = =_ t 
l¢| = min 5 (we tw"), 


the minimum being attained for w = Ww Inspired by this simple relationship, given 
areal number € > 0 and a weight vector w € RY, with w; >0,7 =1,...,N, one 
introduces the functional 


N 
1 
T(z, W,€) = 5 (Guy + ws +07"), zeER, (50) 


j=l 


238 M. Fornasier and H. Rauhut 


The algorithm roughly described in (45) and (46) can be recast as an alternating 
method for choosing minimizers and weights based on the functional .7. To describe 
this more rigorously, recall that r(z) denotes the nonincreasing rearrangement of a 
vector z € RY. 


Algorithm IRLS. Initialize by taking w° := (1,...,1). Set «9 := 1. Then 
recursively define, forn = 0,1,..., 


n+l. : n : 
x = arg min ,w’,é€,) = arg min wn 51 
g min J(w",e) = arg_min Ic G1) 
and 
rKiQx"*") 
€n+1 -= min En, a ae , (52) 
where K is a fixed integer that will be specified later. Set 
witl i= arg min Tl wea). (53) 
w> 


The algorithm stops if €,, = 0; in this case, define x/ := x” for j > n. In general, 
the algorithm generates an infinite sequence (x"),ey of vectors. 


Each step of the algorithm requires the solution of a weighted least squares 
problem. In matrix form 


xttl = D7!A*(AD71A*) ly, (54) 


where D, is the N x N diagonal matrix the j th diagonal entry of which is w’. 


n+1 


Once x”*! is found, the weight w"t! is given by 


witl = [rth + eal J = Degli (55) 


J 


Convergence Properties 
Lemma 3. Set L := J7(x!,w®, e&). Then 


|x” — xP t1I2 < OL [ F(x", w", en) — Ta" wt! en si)]. 
Hence (J(x", w", €n))nen is a monotonically decreasing sequence and 
lim ||x” — xe =0. 
noo 
Proof. Note that J(x",w",€,) => J(x"*t!,w"t!, 6,41) for eachn = 1,2,..., and 


L= IC W eo) > Ia" Wien) > Wit, fH... 
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Hence, for eachn = 1,2,..., the following estimates hold: 


QF (x", Ww", en) — FW enti] 
> PW eae w", En) _ To", w", En)] -_ (en = Pages cae oe 


= ie a antl yn — xt =_ i — xt! yn =n, 5 


N 
— So wi (xt _ ie > Le — ntl? 


j=l 
In the third line, it is used that (x”7!, x” —x"t!) 1 = 0 due to (49) since x” — x"! 
is contained in VV. a 


Moreover, if one assumes that x” — x and e, — 0, then, formally, 
T(x" Ww" €n) > (XII. 


Hence, one expects that this algorithm performs similar to £,-minimization. Indeed, 
the following convergence result holds. 


Theorem 8. Suppose A € R”*% satisfies the NSP of order K with constant y < 1. 
Use K in the update rule (52). Then, for each y € IR”, the sequence x" produced 
by the algorithm converges to a vector X, with rn41(X) = N limy-+o0 €n, and the 
following holds: 


(i) Ife = limypsoo€n = 0, then x is K-sparse; in this case, there is therefore a 
unique €,-minimizer x*, and X = x*; moreover, one has, fork < K and any 


ze F(y), 


lz x|hi 


Pan eae (56) 
l-y 


1/2 
(ii) If € = limy—+o0 €n > 0, then X = x* := arg Mine F(y) eee (2 + <*) : 


(iii) In this last case, if y satisfies the stricter bound y < 1— Co (or, equivalently, 


if oo < K), then one has, for allz € F(y) and anyk < K — roe that 


|z-—X]li < Cog(zZ)1, with é := 


ee 
etn Ss aa eo 


l-y K-k- <4 


As a consequence, this case is excluded if F(y) contains a vector of sparsity 


k<K-<. 
¥ 
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Note that the approximation properties (56) and (57) are exactly of the same order as 
the one (6) provided by £;-minimization. However, in general, x is not necessarily 
an £,-minimizer, unless it coincides with a sparse solution. The proof of this result 
is not included and the interested reader is referred to [44, 69] for the details. 


Rate of Convergence 
It is instructive to show a further result concerning the local rate of convergence 
of this algorithm, which again uses the NSP as well as the optimality conditions 
introduced above. One assumes here that ¥(y) contains the k-sparse vector x*. The 
algorithm produces a sequence x”, which converges to x*, as established above. 
One denotes the (unknown) support of the k-sparse vector x* by T. 

For now, one introduces an auxiliary sequence of error vectors y”" € N via n” := 
x" — x* and 


En = |In" lh = [lx* — x". 


Theorem 8 guarantees that F,, — 0 forn — oo. A useful technical result is reported 
next. 


Lemma 4. For any z,z € RY, and for any j, 
loj@1 —9;@)il < lle-z'lh, (58) 
while for any J > j, 
(J — f)ry(2) < |lz-z/Ih toj@):. (59) 


Proof. To prove (58), approximate z by a best j-term approximation zi A € x; of z’ 
in €;. Then 


oj(2 <z—zlh < llze-Zlh to), 


and the result follows from symmetry. To prove (59), it suffices to note that (J — 
J)ri@) < 0; (21. a 


The following theorem gives a bound on the rate of convergence of E,, to zero. 


Theorem 9. Assume A satisfies the NSP of order K with constant y. Suppose that 
k< Caps l,and0<y< 1 — 4-45 are such that 


ydi+y) 1 
= 1 <i. 
i 1—p ( ay eo 


Assume that F(y) contains a k-sparse vector x* and let T = supp(x*). Let no be 
such that 
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E,, < R* := p min|x;|. (60) 
0 i€T 


Then, for alln = no, one has 
En+i = ME). (61) 
Consequently, x" converges to x* exponentially. 


Proof. The relation (49) with w = w", x” = x"t! = x* + n"*!, and n = x™t! — 
x* = n"*! gives 


N 
pee 4 Atty tly! ”_ 


i=1 


Rearranging the terms and using the fact that x* is supported on 7, one obtains 


n+1)2..n _ * n+l w! a x; n+1 
3 In; [wi = — iain i == 2 [(x")? + eye is (62) 


i=1 i€T 


The proof of the theorem is by induction. Assume that E,, < R* has already been 
established. Then, for alli € T, 


ln? | < |In" Ih = En S plx;, 
so that 


eat el 
[rye + By? = Jar] Ie tnt] ~ 1p 


‘il 


(63) 


and hence (62) combined with (63) and the NSP gives 


1 
+1)2 tel Hel 
ae wi S ltr hs poole I; 


i=1 


The Cauchy—Schwarz inequality combined with the above estimate yields 


iret < (x nth (Xion? + a") 


ieT¢ ieT¢ 
= (> In Pw ‘) (Son - a") 
i=1 bere 


IA 


Toplith (ln la + Nen)- (64) 
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If n7 vt! — 0, then Cie = 0. In this case x”*! is k-sparse and the algorithm has 
stopped by definition; since x”*+! — x* is in the null space NV, which contains no 
eet elements other than 0, one has already obtained the solution x"t! = x*. 
If n'-¢'| # 0, then canceling the factor ||7-7"||1 in (64) yields 
n+ 1 4 n 
Inve lh < Ta" lh + Nen), 
—Pp 
and thus 
n n n 1 may ae 2 
a = leh + dae th s+ ylnge hs (ln lh + Nen). 
(65) 

Now, by (52) and (59), it follows 

Ney Sree") <4 x" xh tore) = Th 6) 

~ ~ K+1-k K+1-k 


since by assumption 0; (x*); = 0. Together with (65), this yields the desired bound: 


En+i = Iln 


‘ae + y) 1+ 
—p K+1-k 


) Wilh = Hey. 


In particular, since 4 < 1, one has E,41 < R*, which completes the induction step. 
It follows that F,4; < WE, for alln > no. |_| 


The linear rate (61) can be improved significantly, by a very simple modification 
of the rule of updating the weight: 


2—t 


wit = (cory? +241) : , jJz=l,...,N, foranyO<1t <1. 
This corresponds to the substitution of the function 7 with 


N 
T 2-—7t | 
Ir (z, w, €) = 5 Zwy +ew; + 2 — |. 


j=l WwW; 


where z € RY we RY, e € R,. With this new up-to-date rule for the weight, 
which depends on 0 < t < 1, one has formally, for x” — x and e, — 0, 


Fe(x" Ww", €n) > Ilx|lz- 


Hence such an iterative optimization tends to promote the ¢,-quasi-norm 
minimization. 
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Surprisingly the rate of local convergence of this modified algorithm is superlin- 
ear; the rate is larger for smaller t and approaches a quadratic rate as t — 0. More 
precisely, the local error E, := ||x” — x*||f satisfies 


En+1 = L(y, areas (67) 


where j4(y, tT) < 1 for y > 0 sufficiently small. The validity of (67) is restricted to 
x” in a (small) ball centered at x*. In particular, if x° is close enough to x*, then 
(67) ensures the convergence of the algorithm to the k-sparse solution x*; see Fig. 4. 


Numerical Experiments 


Figure 5 shows a typical phase transition diagram related to the (experimentally 
determined) probability of successful recovery of sparse vectors by means of the 
iteratively re-weighted least squares algorithm. For each point of this diagram with 
coordinates (m/N,k/m) € [0,1]*, one shows the empirical success probability 
of recovery of a k-sparse vector x € R” from m measurements y = Ax. 
The brightness level corresponds to the probability. As measurement matrix a real 
random Fourier type matrix A was used, with entries given by 


Ax,j = cos(2njé&), jf =1,...,N, 


and the &,k =1,..., m, are sampled independently and uniformly at random from 
[0, 1]. (Theorem 5 does not apply directly to real random Fourier matrices, but an 
analogous result concerning the RIP for such matrices can be found in [131].) 

Figure 6 shows a section of a phase transition diagram related to the (experimen- 
tally determined) probability of successful recovery of sparse vectors from linear 
measurements y = Ax, where the matrix A has i.i.d. Gaussian entries. Here both 
m and WN are fixed and only k is variable. This diagram establishes the transition 
from a situation of exact reconstruction for sparse vectors with high probability 
to very unlikely recovery for vectors with many nonzero entries. These numerical 
experiments used the iteratively re-weighted least squares algorithm with different 
parameters 0 < t < 1. It is of interest to emphasize the enhanced success rate 
when using the algorithm for t < 1. Similarly, many other algorithms are tested 
by showing the corresponding phase transition diagrams and comparing them; see 
[12] for a detailed account of phase transitions for greedy algorithms and [49, 54] 
for £;-minimization. 

This section is concluded by showing applications of £,-minimization methods 
to a real-life image recolorization problem [70,71] in Fig. 7. The image is known 
completely only on very few colored portions, while on the remaining areas, only 
gray levels are provided. With this partial information, the use of £;-minimization 
with respect to wavelet or curvelet coefficients allows for high fidelity recolorization 
of the whole images. 
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Comparison of the rate of convergence for different t 


Logarithmic error 


0 5 10 15 20 25 30 35 40 45 
Number of iterations 


Fig. 4 The decay of logarithmic error is shown, as a function of the number of iterations of IRLS 
for different values of t (1, 0.8, 0.6, 0.56). The results of an experiment are also shown, in which 
the initial 10 iterations are performed with t = 1 and the remaining iterations with t = 0.5 


01 02 03 04 05 06 0.7 08 09 1 


Fig. 5 Empirical success probability of recovery of k-sparse vectors x € R% from measurements 
y = Ax, where A € R”* is a real random Fourier matrix. The dimension N = 300 of the 
vectors is fixed. Each point of this diagram with coordinates (m/N,k/m) € [0, 1]? indicates the 
empirical success probability of recovery, which is computed by running 100 experiments with 
randomly generated k-sparse vectors x and randomly generated matrix. The algorithm used for the 
recovery is the iteratively re-weighted least squares method tuned to promote ¢;-minimization 
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Comparison of iterative re-weighted least squares for |, and |,— |, 
minimization in Compressed Sensing 
1 ;_@_0e_e_e—_¢ © 


—— |, minimization 4 
—e— 1, |, minimization 


Probability of exact reconstruction 
oO 
oa 


0.4+ 4 
0.35 4 
0.24 4 
0.1} 4 
0 o—e 
0 5 10 15 20 25 


k.sparsity 


Fig. 6 Empirical success probability of recovery of a k-sparse vector x € R®° from mea- 
surements y = Ax, where A € IR%*?50 is Gaussian. The matrix is generated once; then, for 
each sparsity value k shown in the plot, 500 attempts were made, for randomly generated k- 
sparse vectors x. Two different IRLS algorithms were compared: one with weights inspired by 
£,-minimization and the IRLS with weights that gradually moved during the iterations from an £,- 
to an £,-minimization goal, with final t = 0.5 


= 


Fig. 7 Iterations of the recolorization methods proposed in [70,71] via ¢, and total variation 
minimization, for the virtual restoration of the frescoes of A. Mantegna (1452), which were 
destroyed by a bombing during World War II. Only a few colored fragments of the images were 
saved from the disaster, together with good quality gray-level pictures dated to 1920 


Extensions to Affine Low-Rank Minimization 


Reformulated as a semidefinite program, the nuclear norm minimization (25) can 
be solved also by general methods from convex optimization [17]. Unfortunately, 
standard semidefinite programming tools work efficiently for solving nuclear norm 
minimization problems only for matrices up to size approximately 100 x 100. 
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Therefore, it is crucial to develop fast algorithms that are specialized to nuclear 
norm minimization (or other heuristics for rank minimization). Beside the already 
mentioned primal-dual algorithm of Sect.4, several alternative approaches have 
been suggested so far; see, for instance, [20, 85, 97,98, 104]. 

Borrowing a leaf from iteratively re-weighted least squares for £,-minimization, 
this section discusses an algorithm inspired by (25). Assume X € M,x,,0~< W = 
W* &€ M,xn, and consider the functional, 


1 = 
T(XW) = 5 (IWIOX |e + WIE), (68) 


where || - || ¢ is the Frobenius norm. In order to define the iteration below, one recalls 
that o;, (X) denotes the k th singular value of a matrix X. 


IRLS-M algorithm for low-rank matrix recovery: Initialize by taking W° := 
IT € Myxn. Set €9 := 1, K > k, and y > 0. Then recursively define, for 2 = 
12335 
X¢:=arg min J(X,W) 
S(X)=M 
and 
€¢ := min {ee-1, yox+i(X")} : (69) 


The update of the weight matrix W‘ follows the variational principle 


W° := arg min I (X',W) (70) 
0<W=W* xe, 'T 


The algorithm stops if eg = 0; in this case, define X/ := X° for j > ¢. In 
general, the algorithm generates an infinite sequence (X‘)en of matrices. 


The following convergence result can be shown with an analysis similar to the 
one done for the IRLS for sparse vector recovery, under the assumption that the 
measurement map .” satisfies the restricted isometry property in Definition 4. One 
refers to [73] for details. 


Proposition 2. Consider the IRLS-M algorithm with parameters y = 1/nand K € 
N. Let Y : Mixp — C” be a surjective map with restricted isometry constants 
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=. 1- Zs: Then, if there exists a k-rank matrix X 


63x, O4x Satisfying n = 
satisfying S(X) = M withk < K — ae the sequence (X")ven converges to X. 


Actually, one can prove something stronger: the IRLS-M algorithm is robust, 
in the sense that under the same conditions on the measurement map .”, the 
accumulation points of the IRLS-M algorithm are guaranteed to approximate an 
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arbitrary X € M, x» from the measurements .@ = .#(X) to within a factor of the 
best k-rank approximation error of X in the nuclear norm. 

By combining both IRLS and IRLS-M, one can formulate an iteratively re- 
weighted least squares to solve also mixed problems of the type (35). 


5 Open Questions 


The field of compressed sensing is rather young, so there remain many directions to 
be explored, and it is impossible to give an exhaustive list here. Below, we focus on 
two problems which seem to be rather hard and remained unsolved until the time of 
writing of this article. 


Deterministic Compressed Sensing Matrices 


So far, only several types of random matrices A €¢ C”*" 


RIP 6, < 6 < 0.4 (say) for 


are known to satisfy the 


m = Css log*(N) (71) 


for some constant Cs and some exponent @ (with high probability). This is a 
strong form of existence statement. It is open, however, to provide deterministic 
and explicit m x N matrices that satisfy the RIP 6, < 6 < 0.5 (say) in the desired 
range (71). 

In order to show RIP estimates in the regime (71), one has to take into account 
cancellations of positive and negative (or more generally complex) entries in the 
matrix; see also Sect. 3. This is done “automatically” with probabilistic methods but 
seems to be much more difficult to exploit when the given matrix is deterministic. 
It may be conjectured that certain equiangular tight frames or the “Alltop matrix” in 
[123,143] do satisfy the RIP under (71). This is supported by numerical experiments 
in [123]. It is expected, however, that a proof is very hard and requires a good 
amount of analytic number theory. An involved deterministic construction that 
achieves the RIP in the regime m > Cs?~* — overcoming the quadratic bottleneck 
inherent to bounds via the coherence — was provided in [15, 16], but the best 
available estimates of ¢ are in the range of 10~76 [112]. One refers to [76, Chapter 
6] for more background information on deterministic RIP matrices. 

Another approach for deterministic constructions of CS matrices uses determin- 
istic expander graphs [11]. Instead of the usual RIP, one shows that the adjacency 
matrix of such an expander graph has the 1-RIP, where the £2-norm is replaced 
by the £;-norm at each occurrence in (8). The 1-RIP also implies recovery by £- 
minimization. The best known deterministic expanders [33] yield sparse recovery 
under the condition m > Cs(log N)° log?(N) | Although the scaling in s is linear 
as desired, the term (log NV)‘ log*(N) grows faster than any polynomial in log NV. 
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Another drawback is that the deterministic expander graph is the output of a 
polynomial time algorithm, and it is questionable whether the resulting matrix can 
be regarded as explicit. 


Removing Log-Factors in the Fourier-RIP Estimate 


It is known [26, 76, 129, 131, 138] that a random partial Fourier matrix A € Cmxy 
satisfies the RIP with high probability provided 


—— > Css log’(s) log(N). 
log(m) 


(The condition stated in (19) implies this one.) It is conjectured that one can remove 
some of the log-factors. It must be hard, however, to improve this to a better estimate 
than m > Css log(N) log(log NV). Indeed, this would imply an open conjecture 
of Talagrand [146] concerning the equivalence of the €; and £2 norm of a linear 
combination of a subset of characters (complex exponentials). One refers to [76, 
Chapter 12.7] for more details. 


Compressive Sensing with Nonlinear Measurements 


As described in Sect.3, the extension of compressed sensing towards nonlinear 
measurements is linked through the solution of problems of the type (35) which 
lead to semidefinite programming. Although some progress has been achieved, 
there is still the need to develop highly efficient algorithms for solving such 
problems in large dimensions. Besides these more practical aspects, also theoretical 
guarantees for algorithms to stably recover nearly sparse solutions consistent with 
given nonlinear measurements, for instance, phase retrieval problems beyond the 
case of Gaussian measurements, are a current field of very active research. 


6 Conclusion 


Compressive sensing established itself by now as a new sampling theory which 
exhibits fundamental and intriguing connections with several mathematical fields, 
such as probability, geometry of Banach spaces, harmonic analysis, theory of com- 
putability, and information-based complexity. The link to convex optimization and 
the development of very efficient and robust numerical methods make compressive 
sensing a concept useful for a broad spectrum of natural science and engineering 
applications, in particular, in signal and image processing and acquisition. It can 
be expected that compressive sensing will enter various branches of science and 
technology to notable effect. 
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New challenges are now emerging in numerical analysis and simulation where 
high-dimensional problems (e.g., stochastic partial differential equations in finance 
and electron structure calculations in chemistry and biochemistry) became the 
frontier. In this context, besides other forms of efficient approximation, such as 
sparse grid and tensor product methods [18], compressive sensing is a promising 
concept which is likely to cope with the “curse of dimensionality.” In particular, 
further systematic developments of adaptivity in the presence of different scales, 
randomized algorithms, and an increasing role for combinatorial aspects of the 
underlying algorithms are examples of possible future developments, which are 
inspired by the successful history of compressive sensing [145]. 


Cross-References 


Compressive sensing has connections with the following chapters of the book: 


Duality and Convex Programming 

Gabor Analysis for Imaging 

Large-Scale Inverse Problems in Imaging 

Linear Inverse Problems 

Mumford and Shah Model and its Applications to Image Segmentation and Image 
Restoration 

Numerical Methods and Applications in Total Variation Image Restoration 
Regularization Methods for Ill-Posed Problems 

Sampling Methods 

Splines and Multiresolution Analysis 

Starlet Transform in Astronomical Data Processing 

Supervised Learning by Support Vector Machines 

Synthetic Aperture Radar Imaging 

Tomography 

Total Variation in Imaging 


Recommended Reading 


The initial papers on the subject are [26, 29, 47]. The monograph [76] provides an 
introduction to compressive sensing. Further introductory articles can be found in 
[6, 21,28, 61, 63, 69, 131, 136]. 


References 


1. Achlioptas, D.: | Database-friendly random projections. In: Proceedings of the 20th 
Annual ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 
Santa Barbara, pp. 274-281 (2001) 


250 


27. 


28. 


M. Fornasier and H. Rauhut 


. Affentranger, F., Schneider, R.: Random projections of regular simplices. Discret. Comput. 


Geom. 7(3), 219-226 (1992) 


. Ailon, N., Liberty, E.: Almost optimal unrestricted fast Johnson-Lindenstrauss transform. In: 


Symposium on Discrete Algorithms (SODA), San Francisco, (2011) 


. Alexeev, B., Bandeira, A.S., Fickus, M., Mixon, D.G.: Phase retrieval with polarization 


(2012). arXiv:1210.7752 


. Balan, R., Casazza, P., Edidin, D.: On signal reconstruction without phase. Appl. Comput. 


Harmon. Anal. 20(3), 345-356 (2006) 


. Baraniuk, R.: Compressive sensing. IEEE Signal Process. Mag. 24(4), 118-121 (2007) 
. Baraniuk, R.G., Davenport, M., DeVore, R.A., Wakin, M.: A simple proof of the restricted 


isometry property for random matrices. Constr. Approx. 28(3), 253-263 (2008) 


. Bauschke, H.H., Combettes, P.-L., Luke, D.R.: Hybrid projection-reflection method for phase 


retrieval. J. Opt. Soc. Am. A 20(6), 1025—1034 (2003) 


. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse 


problems. SIAM J. Imaging Sci. 2(1), 183-202 (2009) 


. Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image 


denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419-2434 (2009) 


. Berinde, R., Gilbert, A.C., Indyk, P., Karloff, H., Strauss, M.: Combining geometry and 


combinatorics: a unified approach to sparse signal recovery. In: Proceedings of the 46th 
Annual Allerton Conference on Comunication, Control, and Computing 2008, Urbana, 
pp. 798-805. IEEE (2008) 


. Blanchard, J.D., Cartis, C., Tanner, J.. Thompson, A.: Phase transitions for greedy sparse 


approximation algorithms. Appl. Comput. Harmon. Anal. 30(2), 188-203 (2011) 


. Blumensath, T., Davies, M.: Iterative hard thresholding for compressed sensing. Appl. 


Comput. Harmon. Anal. 27(3), 265-274 (2009) 


. Bobin, J., Starck, J.-L., Ottensamer, R.: Compressed sensing in astronomy. IEEE J. Sel. Top. 


Signal Process. 2(5), 718-726 (2008) 


. Bourgain, J., Dilworth, S., Ford, K., Konyagin, S., Kutzarova, D.: Breaking the k?-barrier for 


explicit RIP matrices. In: STOC’11, San Jose, pp. 637-644 (2011) 


. Bourgain, J., Dilworth, S., Ford, K., Konyagin, S., Kutzarova, D.: Explicit constructions of 


RIP matrices and related problems. Duke Math. J. 159(1), 145-185 (2011) 


. Boyd, S., Vandenberghe, L.: | Convex Optimization. | Cambridge University Press, 


Cambridge/New York (2004) 


. Bungartz, H.-J., Griebel, M.: Sparse grids. Acta Numer. 13, 147-269 (2004) 
. Cai, T., Zhang, A.: Sparse representation of a polytope and recovery of sparse signals and 


low-rank matrices. IEEE Trans. Inf. Theory 60(1), 122-132 (2014) 


. Cai, J.-F, Candés, E.J., Shen, Z.: A singular value thresholding algorithm for matrix 


completion. SIAM J. Optim. 20(4), 1956-1982 (2010) 


. Candés, E.J.: Compressive sampling. In: Proceedings of the International Congress of 


Mathematicians, Madrid (2006) 


. Candés, E.J.: The restricted isometry property and its implications for compressed sensing. 


C.R. Acad. Sci. Paris Ser. I Math. 346, 589-592 (2008) 


. Candeés, E.J., Li, X.: Solving quadratic equations via PhaseLift when there are about as many 


equations as unknowns. Found. Comput. Math. 14(5), 1017-1026 (2014) 


. Candés, E.J., Plan, Y.: Tight oracle bounds for low-rank matrix recovery from a minimal 


number of random measurements. IEEE Trans. Inf. Theory 57(4), 2342-2359 (2011) 


. Candeés, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. 


Math. 9, 717-772 (2009) 


. Candés, E.J., Tao, T.: Near optimal signal recovery from random projections: universal 


encoding strategies? IEEE Trans. Inf. Theory 52(12), 5406-5425 (2006) 

Candés, E.J., Tao, T.: The power of convex relaxation: near-optimal matrix completion. IEEE 
Trans. Inf. Theory 56(5), 2053-2080 (2010) 

Candés, E., Wakin, M.: An introduction to compressive sampling. IEEE Signal Process. Mag. 
25(2), 21-30 (2008) 


Compressive Sensing 251 


29. 


30. 


31. 


32. 


33. 


34. 


35. 


36. 


37. 
38. 


39. 


40. 


41. 


42. 


43. 


44. 


45. 


46. 


47. 
48. 


49. 


50. 


51. 


52. 


53: 


Candés, E.J., Tao, T., Romberg, J.: Robust uncertainty principles: exact signal reconstruction 
from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489-509 
(2006) 

Candés, E.J., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate 
measurements. Commun. Pure Appl. Math. 59(8), 1207-1223 (2006) 

Candés, E., Li, X., Soltanolkotabi, M.: Phase retrieval from masked Fourier transforms (2013, 
preprint) 

Candés, E.J., Strohmer, T., Voroninski, V.: PhaseLift: exact and stable signal recovery from 
magnitude measurements via convex programming. Commun. Pure Appl. Math. 66(8), 1241- 
1274 (2013) 

Capalbo, M., Reingold, O., Vadhan, S., Wigderson, A.: Randomness conductors and constant- 
degree lossless expanders. In: Proceedings of the Thirty-Fourth Annual ACM Symposium on 
Theory of Computing, Montréal, pp. 659-668 (electronic). ACM (2002) 

Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with 
applications to imaging. J. Math. Imaging Vis. 40, 120-145 (2011) 

Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. 
Sci. Comput. 20(1), 33-61 (1999) 

Christensen, O.: An Introduction to Frames and Riesz Bases. Applied and Numerical 
Harmonic Analysis. Birkhauser, Boston (2003) 

Cline, A.K.: Rate of convergence of Lawson’s algorithm. Math. Comput. 26, 167-176 (1972) 
Cohen, A., Dahmen, W., DeVore, R.A.: Compressed sensing and best k-term approximation. 
J. Am. Math. Soc. 22(1), 211-231 (2009) 

Combettes, P., Pesquet, J.-C.: A Douglas-Rachford splitting approach to nonsmooth convex 
variational signal recovery. IEEE J. Sel. Top. Signal Process. 1(4), 564-574 (2007) 
Combettes, P., Pesquet, J.-C.: Proximal splitting methods in signal processing. In: 
Bauschke, H., Burachik, R., Combettes, P., Elser, V., Luke, D., Wolkowicz, H. (eds.) Fixed- 
Point Algorithms for Inverse Problems in Science and Engineering, pp. 185-212. Springer, 
New York (2011) 

Combettes, P., Wajs, V.: Signal recovery by proximal forward-backward splitting. Multisc. 
Model. Simul. 4(4), 1168-1200 (electronic) (2005) 

Cormode, G., Muthukrishnan, S.: Combinatorial algorithms for compressed sensing. In: 
CISS, Princeton (2006) 

Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse 
problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413-1457 (2004) 
Daubechies, I., DeVore, R., Fornasier, M., Giintiirk, C.: Iteratively re-weighted least squares 
minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1-38 (2010) 

Davies, M., Gribonval, R.: Restricted isometry constants where £” sparse recovery can fail 
for 0 < p < 1. IEEE Trans. Inf. Theory 55(5), 2203-2214 (2009) 

Do, B., Indyk, P., Price, E., Woodruff, D.: Lower bounds for sparse recovery. In: Proceedings 
of the SODA, Austin (2010) 

Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289-1306 (2006) 
Donoho, D.L.: For most large underdetermined systems of linear equations the minimal /! 
solution is also the sparsest solution. Commun. Pure Appl. Anal. 59(6), 797-829 (2006) 
Donoho, D.L.: High-dimensional centrally symmetric polytopes with neighborliness propor- 
tional to dimension. Discret. Comput. Geom. 35(4), 617-652 (2006) 

Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) 
dictionaries via £! minimization. Proc. Natl. Acad. Sci. USA 100(5), 2197-2202 (2003) 
Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decompositions. IEEE Trans. 
Inf. Theory 47(7), 2845-2862 (2001) 

Donoho, D., Logan, B.: Signal recovery and the large sieve. SIAM J. Appl. Math. 52(2), 
577-591 (1992) 

Donoho, D.L., Tanner, J.: | Neighborliness of randomly projected simplices in high 
dimensions. Proc. Natl. Acad. Sci. USA 102(27), 9452-9457 (2005) 


252 


54 


D0: 


56. 


57. 


58. 


59. 


60. 


61. 


62. 


63. 


64. 


65. 
66. 


67. 


68. 
69. 


70. 


71. 


72. 


73. 


74. 


75. 


76. 


77. 


2. 


79. 


M. Fornasier and H. Rauhut 


. Donoho, D.L., Tanner, J.: | Counting faces of randomly-projected polytopes when the 
projection radically lowers dimension. J. Am. Math. Soc. 22(1), 1-53 (2009) 

Donoho, D.L., Tsaig, Y.: Fast solution of 11-norm minimization problems when the solution 
may be sparse. IEEE Trans. Inf. Theory 54(11), 4789-4812 (2008) 

Dorfman, R.: The detection of defective members of large populations. Ann. Stat. 14, 436- 
440 (1943) 

Douglas, J., Rachford, H.: On the numerical solution of heat conduction problems in two or 
three space variables. Trans. Am. Math. Soc. 82, 421-439 (1956) 

Duarte, M., Davenport, M., Takhar, D., Laska, J., Ting, S., Kelly, K., Baraniuk, R.: Single- 
pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83-91 (2008) 
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32(2), 
407-499 (2004) 

Ehler, M., Fornasier, M., Sigl, J.: Quasi-linear compressed sensing. Multiscale Model. Simul. 
12(2), 725-754 (2014) 

Elad, M.: Sparse and Redundant Representations: From Theory to Applications in Signal and 
Image Processing. Springer, New York (2010) 

Elad, M., Bruckstein, A.M.: A generalized uncertainty principle and sparse representation in 
pairs of bases. IEEE Trans. Inf. Theory 48(9), 2558-2567 (2002) 

Eldar, Y., Kutyniok, G. (eds.): Compressed Sensing — Theory and Applications. Cambridge 
University Press, Cambridge/New York (2012) 

Eldar, Y., Mendelson, S.: Phase retrieval: stability and recovery guarantees. Appl. Comput. 
Harmon. Anal. (to appear). doi:10.1016/j.acha.2013.08.003 

Ender, J.: On compressive sensing applied to radar. Signal Process. 90(5), 1402-1414 (2010) 
Fannjiang, A., Yan, P., Strohmer, T.: Compressed remote sensing of sparse objects. SIAM J. 
Imaging Sci. 3(3), 595-618 (2010) 

Fazel, M.: Matrix rank minimization with applications. PhD thesis, Stanford University 
(2002) 

Fienup, J.R.: Phase retrieval algorithms: a comparison. Appl. Opt. 21(15), 2758-2769 (1982) 
Fornasier, M.: Numerical methods for sparse recovery. In: Fornasier, M. (ed.) Theoretical 
Foundations and Numerical Methods for Sparse Recovery. Radon Series on Computational 
and Applied Mathematics, vol. 9, pp. 93-200. deGruyter, Berlin (2010). Papers based on the 
presentations of the summer school “Theoretical Foundations and Numerical Methods for 
Sparse Recovery”, Vienna, Austria, 31 Aug-4 Sept 2009 

Fornasier, M., March, R.: Restoration of color images by vector valued BV functions and 
variational calculus. SIAM J. Appl. Math. 68(2), 437-460 (2007) 

Fornasier, M., Ramlau, R., Teschke, G.: The application of joint sparsity and total variation 
minimization algorithms to a real-life art restoration problem. Adv. Comput. Math. 31(1-3), 
157-184 (2009) 

Fornasier, M., Langer, A., Schénlieb, C.: A convergent overlapping domain decomposition 
method for total variation minimization. Numer. Math. 116(4), 645-685 (2010) 

Fornasier, M., Rauhut, H., Ward, R.: Low-rank matrix recovery via iteratively reweighted 
least squares minimization. SIAM J. Optim. 21(4), 1614-1640 (2011) 

Foucart, S.: A note on guaranteed sparse recovery via €;-minimization. Appl. Comput. 
Harmon. Anal. 29(1), 97-103 (2010) 

Foucart, S., Lai, M.: | Sparsest solutions of underdetermined linear systems via ¢,- 
minimization for 0 < g < 1. Appl. Comput. Harmon. Anal. 26(3), 395-407 (2009) 

Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Applied and 
Numerical Harmonic Analysis. Birkhauser, Boston (2013) 

Foucart, S., Pajor, A., Rauhut, H., Ullrich, T.: The Gelfand widths of £,-balls for 0 < p < 1. 
J. Complex. 26(6), 629-640 (2010) 

Fuchs, J.J.: On sparse representations in arbitrary redundant bases. IEEE Trans. Inf. Theory 
50(6), 1341-1344 (2004) 

Garnaev, A., Gluskin, E.: On widths of the Euclidean ball. Sov. Math. Dokl. 30, 200-204 
(1984) 


Compressive Sensing 253 


80. 


81. 


82. 


83. 
84. 
85. 
86. 
87. 
88. 
89. 
90. 
91. 
92. 
93. 
94. 
95. 
96. 
97. 
98. 
99. 
100. 
101. 
102. 


103. 
. Lee, K., Bresler, Y.: ADMiRA: atomic decomposition for minimum rank approximation. 


104 


Gilbert, A.C., Muthukrishnan, S., Guha, S., Indyk, P., Strauss, M.: Near-optimal sparse 
Fourier representations via sampling. In: Proceedings of the STOC’02, Montréal, pp. 152- 
161. Association for Computing Machinery (2002) 

Gilbert, A.C., Muthukrishnan, S., Strauss, M.J.: Approximation of functions over redundant 
dictionaries using coherence. In: Proceedings of the Fourteenth Annual ACM-SIAM 
Symposium on Discrete Algorithms, Baltimore, 12-14 Jan 2003, pp. 243-252. SIAM and 
Association for Computing Machinery (2003) 

Gilbert, A.C., Strauss, M., Tropp, J.A., Vershynin, R.: One sketch for all: fast algorithms for 
compressed sensing. In: Proceedings of the 39th ACM Symposium Theory of Computing 
(STOC), San Diego (2007) 

Glowinski, R., Le, T.: Augmented Lagrangian and Operator-Splitting Methods. SIAM, 
Philadelphia (1989) 

Gluskin, E.: Norms of random matrices and widths of finite-dimensional sets. Math. 
USSR-Sb. 48, 173-182 (1984) 

Goldfarb, D., Ma, S.: Convergence of fixed point continuation algorithms for matrix rank 
minimization. Found. Comput. Math. 11(2), 183-210 (2011) 

Gorodnitsky, I., Rao, B.: Sparse signal reconstruction from limited data using FOCUSS: a 
re-weighted minimum norm algorithm. IEEE Trans. Signal Process. 45(3), 600-616 (1997) 
Gribonval, R., Nielsen, M.: Sparse representations in unions of bases. IEEE Trans. Inf. 
Theory 49(12), 3320-3325 (2003) 

Gross, D.: Recovering low-rank matrices from few coefficients in any basis. IEEE Trans. Inf. 
Theory 57(3), 1548-1566 (2011) 

Gross, D., Liu, Y.-K., Flammia, S.T., Becker, S., Eisert, J.: Quantum state tomography via 
compressed sensing. Phys. Rev. Lett. 105, 150401 (2010) 

Gross, D., Krahmer, F., Kueng, R.: Improved recovery guarantees for phase retrieval from 
coded diffraction patterns (2014, preprint) 

Gross, D., Krahmer, F., Kueng, R.: A partial derandomization of PhaseLift using spherical 
designs. J. Fourier Anal. Appl. (to appear) 

He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: 
from contraction perspective. SIAM J. Imaging Sci. 5(1), 119-149 (2012) 

Horn, R., Johnson, C.: Matrix Analysis. Cambridge University Press, Cambridge/New York 
(1990) 

Hiigel, M., Rauhut, H., Strohmer, T.: Remote sensing via 11-minimization. Found. Comput. 
Math. 14, 115-150 (2014) 

Johnson, W.B., Lindenstrauss, J. (eds.): Handbook of the Geometry of Banach Spaces, vol. I. 
North-Holland, Amsterdam (2001) 

Kashin, B.: Diameters of some finite-dimensional sets and classes of smooth functions. Math. 
USSR Izv. 11, 317-333 (1977) 

Keshavan, R.H., Montanari, A., Oh, S.: Matrix completion from a few entries. [EEE Trans. 
Inf. Theory 56, 2980-2998 (2010) 

Keshavan, R.H., Montanari, A., Oh, S.: Matrix completion from noisy entries. J. Mach. 
Learn. Res. 11, 2057-2078 (2010) 

Krahmer, F., Rauhut, H.: Structured random measurements in signal processing. GAMM 
Mitteilungen. (to appear) 

Krahmer, F., Ward, R.: New and improved Johnson-Lindenstrauss embeddings via the 
restricted isometry property. SIAM J. Math. Anal. 43(3), 1269-1281 (2011) 

Krahmer, F., Mendelson, S., Rauhut, H.: Suprema of chaos processes and the restricted 
isometry property. Commun. Pure Appl. Math. (to appear). doi:10.1002/cpa.21504 

Lawson, C.: Contributions to the theory of linear least maximum approximation. PhD thesis, 
University of California, Los Angeles (1961) 

Ledoux, M., Talagrand, M.: Probability in Banach Spaces. Springer, Berlin/New York (1991) 


IEEE Trans. Inf. Theory 56(9), 4402-4416 (2010) 


254 


105. 


106. 
107. 


108. 


109. 


110. 


111. 


112. 


113. 


114. 


115. 


116. 


117. 


118. 


119. 


120. 


121. 


122. 


123. 


124. 


125. 


126. 


127. 


128. 


129. 


M. Fornasier and H. Rauhut 


Li, X., Voroninski, V.: Sparse signal recovery from quadratic measurements via convex 
programming (2013). arXiv:1209.4785 

Logan, B.: Properties of high-pass signals. PhD thesis, Columbia University (1965) 
Lorentz, G.G., von Golitschek, M., Makovoz, Y.: Constructive Approximation: Advanced 
Problems. Springer, Berlin (1996) 

Mallat, S.G., Zhang, Z.: Matching pursuits with time-frequency dictionaries. [EEE Trans. 
Signal Process. 41(12), 3397-3415 (1993) 

Marple, S.: Digital Spectral Analysis with Applications. Prentice-Hall, Englewood Cliffs 
(1987) 

Mendelson, S., Pajor, A., Tomcezak Jaegermann, N.: Uniform uncertainty principle for 
Bernoulli and subgaussian ensembles. Constr. Approx. 28(3), 277-289 (2009) 

Millane, R.: Phase retrieval in crystallography and optics. J. Opt. Soc. Am. A 7(3), 394-411 
(1990) 

Mixon, D.: Short, fat matrices. Blog (2013) 

Mohan, K., Fazel, M.: Reweighted nuclear norm minimization with application to system 
identification. In: Proceedings of the American Control Conference, Baltimore, pp. 2953- 
2959 (2010) 

Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227- 
234 (1995) 

Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Program- 
ming. Volume 13 of SIAM Studies in Applied Mathematics. Society for Industrial and 
Applied Mathematics (SIAM), Philadelphia (1994) 

Netrapalli, P., Jain, P., Sanghavi, S.: Phase retrieval using alternating minimization (2013). 
arXiv: 1306.0160 

Novak, E.: Optimal recovery and n-widths for convex classes of functions. J. Approx. Theory 
80(3), 390-408 (1995) 

Ohlsson, H., Yang, A.Y., Dong, R., Sastry, S.S.: Nonlinear basis pursuit. In: 47th Asilomar 
Conference on Signals, Systems and Computers, Pacific Grove (2013) 

Osborne, M., Presnell, B., Turlach, B.: A new approach to variable selection in least squares 
problems. IMA J. Numer. Anal. 20(3), 389-403 (2000) 

Osborne, M., Presnell, B., Turlach, B.: On the LASSO and its dual. J. Comput. Graph. Stat. 
9(2), 319-337 (2000) 

Oymak, S., Mohan, K., Fazel, M., Hassibi, B.: A simplified approach to recovery conditions 
for low-rank matrices. In: Proceedings of the IEEE International Symposium on Information 
Theory (ISIT), St. Petersburg (2011) 

Pfander, G.E., Rauhut, H.: Sparsity in time-frequency representations. J. Fourier Anal. Appl. 
16(2), 233-260 (2010) 

Pfander, G.E., Rauhut, H., Tanner, J.: Identification of matrices having a sparse representation. 
IEEE Trans. Signal Process. 56(11), 5376-5388 (2008) 

Pfander, G.E., Rauhut, H., Tropp, J.A.: The restricted isometry property for time-frequency 
structured random matrices. Probab. Theory Relat. Fields 156, 707-737 (2013) 

Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms 
in convex optimization. In: IEEE International Conference Computer Vision (ICCV), 
Barcelona, pp. 1762-1769 (2011) 

Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the 
Mumford-Shah functional. In: ICCV Proceedings, Kyoto. Springer (2009) 

Prony, R.: Essai expérimental et analytique sur les lois de la Dilatabilité des 
uides élastique et sur celles de la Force expansive de la vapeur de lAoeau et de 
la vapeur de l’alkool, A différentes températures. J. Ecole Polytechnique 1, 24-76 
(1795) 

Rauhut, H.: Random sampling of sparse trigonometric polynomials. Appl. Comput. Harmon. 
Anal. 22(1), 16-42 (2007) 

Rauhut, H.: Stability results for random sampling of sparse trigonometric polynomials. IEEE 
Trans. Inf Theory 54(12), 5661-5670 (2008) 


Compressive Sensing 255 


130. 


131. 


132. 
133. 


134. 


135. 


136. 


137. 


138. 


139. 


140. 


141. 


142. 


143. 


144. 


145. 


146. 


147. 


148. 


149. 


150. 


151. 


152. 


153. 


154. 


Rauhut, H.: Circulant and Toeplitz matrices in compressed sensing. In: Proceedings of the 
SPARS’09 (2009) 

Rauhut, H.: Compressive sensing and structured random matrices. In: Fornasier, M. (ed.) 
Theoretical Foundations and Numerical Methods for Sparse Recovery. Radon Series on 
Computational and Applied Mathematics, vol. 9, pp. 1-92. deGruyter, Berlin (2010). Papers 
based on the presentations of the summer school “Theoretical Foundations and Numerical 
Methods for Sparse Recovery”, Vienna, Austria, 31 Aug-4 Sept 2009 

Rauhut, H., Ward, R.: Interpolation via weighted 11 minimization (2013). ArXiv:1308.0759 
Rauhut, H., Romberg, J.K., Tropp, J.A.: Restricted isometries for partial random circulant 
matrices. Appl. Comput. Harmon. Anal. 32(2), 242-254 (2012) 

Recht, B.: A simpler approach to matrix completion. J. Mach. Learn. Res. 12, 3413-3430 
(2012) 

Recht, B., Fazel, M., Parrilo, P.: Guaranteed minimum-rank solutions of linear matrix 
equations via nuclear norm minimization. SIAM Rev. 52(3), 471-501 (2010) 

Romberg, J.: Imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 14-20 
(2008) 

Romberg, J.K.: Compressive sensing by random convolution. SIAM J. Imaging Sci. 2(4), 
1098-1128 (2009) 

Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measure- 
ments. Commun. Pure Appl. Math. 61, 1025-1045 (2008) 

Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. 
Physica D 60(1-4), 259-268 (1992) 

Santosa, F., Symes, W.: Linear inversion of band-limited reflection seismograms. SIAM J. 
Sci. Stat. Comput. 7(4), 1307-1330 (1986) 

Schnass, K., Vandergheynst, P.: Dictionary preconditioning for greedy algorithms. IEEE 
Trans. Signal Process. 56(5), 1994-2002 (2008) 

Starck, J.-L., Murtagh, F., Fadili, J.: Sparse Image and Signal Processing Wavelets, Curvelets, 
Morphological Diversity, xvii, p. 316. Cambridge University Press, Cambridge (2010) 
Strohmer, T., Heath, R.W., Jr: Grassmannian frames with applications to coding and 
communication. Appl. Comput. Harmon. Anal. 14(3), 257-275 (2003) 

Strohmer, T., Hermann, M.: Compressed sensing radar. In: IEEE Proceedings of the 
International Conference on Acoustic, Speech, and Signal Processing, Las Vegas, pp. 1509- 
1512 (2008) 

Tadmor, E.: Numerical methods for nonlinear partial differential equations. In: Meyers, R.A. 
(ed.) Encyclopedia of Complexity and Systems Science. Springer, New York/London (2009) 
Talagrand, M.: Selecting a proportion of characters. Isr. J. Math. 108, 173-191 (1998) 
Taubock, G., Hlawatsch, F., Eiwen, D., Rauhut, H.: Compressive estimation of doubly 
selective channels in multicarrier systems: leakage effects and sparsity-enhancing processing. 
IEEE J. Sel. Top. Signal Process. 4(2), 255-271 (2010) 

Taylor, H., Banks, S., McCoy, J.: Deconvolution with the £;-norm. Geophys. J. Int. 44(1), 
39-52 (1979) 

Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58(1), 
267-288 (1996) 

Traub, J., Wasilkowski, G., Wo’zniakowski, H.: Information-Based Complexity. Computer 
Science and Scientific Computing. Academic, Boston (1988) 

Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. 
Theory 50(10), 2231-2242 (2004) 

Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. 
IEEE Trans. Inf. Theory 51(3), 1030-1051 (2006) 

Tropp, J., Needell, D.: CoSaMP: iterative signal recovery from incomplete and inaccurate 
samples. Appl. Comput. Harmon. Anal. 26(3), 301-321 (2009) 

Tropp, J.A., Laska, J.N., Duarte, M.F., Romberg, J.K., Baraniuk, R.G.: Beyond nyquist: 
efficient sampling of sparse bandlimited signals. IEEE Trans. Inf. Theory 56(1), 520-544 
(2010) 


256 


155. 
156. 


157. 
158. 


159. 


160. 


M. Fornasier and H. Rauhut 


Unser, M.: Sampling—S0 years after Shannon. Proc. IEEE 88(4), 569-587 (2000) 

van den Berg, E., Friedlander, M.: Probing the Pareto frontier for basis pursuit solutions. 
SIAM J. Sci. Comput. 31(2), 890-912 (2008) 

Vybiral, J.: Widths of embeddings in function spaces. J. Complex. 24(4), 545-570 (2008) 
Wagner, G., Schmieder, P., Stern, A., Hoch, J.: Application of non-linear sampling schemes 
to cosy-type spectra. J. Biomol. NMR 3(5), 569 (1993) 

Willett, R., Marcia, R., Nichols, J.: Compressed sensing for practical optical imaging systems: 
a tutorial. Opt. Eng. 50(7), 072601-072601-13 (2011) 

Willett, R., Duarte, M., Davenport, M., Baraniuk, R.: Sparsity and structure in hyperspectral 
imaging: sensing, reconstruction, and target detection. IEEE Signal Proc. Mag. 31(1), 116- 
126 (2014) 


Duality and Convex Programming 


Jonathan M. Borwein and D. Russell Luke 


Contents 


DV, bOI es cseis.ce said eakawmseedeoratabsdcdeventenates renner andsrcmnoakamudente tame ssadeatiedes 


Forae ime with: Missine Aba... cc. cncmeae ctemdasangmsnice arene siieesitnnas au onemasiimentimenaries 
Imace Denoising and DeconvolitiOn..c...ccsicc ice ectinescn asics eiee nea shadagenaigdes 
TROIS Se NER os ccs daar Sanne cstndatinpaatadenaasenad hatemintiensaealgacume ri noamtes 
Fredholm Integral Equations: o.cscccis geewine arate silane cvs cectumae hese ntadiewneiadeaben’s 
BT BAIN sass Src cass oe ane cae assay asa sta cha eta daselakarpealaalanera teajardenanajaiauinaie atiiarals 
Deapsehintzteay Proper es a a.ciniicsie on csape scene sian yarn a Bare sre cperaceys aid @ eyeroentiaiedeeniel omleamneenaraabnieice 
SUM ALIMPCMM AUS ccd ys Sy sity « shatdleceraaare dt Maiaite abaceakietantetedins st byalen Since meena aeaninate’ 
2 Dishing aie CC oayek Ais 6 tcnicc tea daiennacatiandacana vend catesmadeas eawdos gad odetadeags oe 
PRMCME! COMICAL. gcc. avi wave scowl odiaseeelanie atieivs. weielsminig e aWis hia aide wales plone sh /aieiohy sue 
Bia Fae AU sees aacsiosians caine oacaaiei oe eagettamnatc ealarasaiata ge Mioteatediieateniely aatadalag sig doataeiamicls 
PP UCAMOIS i vids cciren cer ndiing ancien ammedeiemnghepannaiucearnsmnae legal anienecmse atin aie 
Optimality-and Laveange Miiltipliets... ce ciscscu. cody. cesweciesaiunse ebine oueylensions deenreee’s 
Variaotial Printa ples: 5 os cccsacacascnedaaaeds ev aoeie nid teadiee denned caaadeaaseaenenmeageek. 
Fixed Point Theory and Monotome Operators: ........0.0ccccseneseseeeaneneeamsqeennges ees 
NCAR a ETMCRUID I oi sits oocojaloce Santa osha ininlarganbinilersral iashiase rash itinlgscraatoso aie tedlaiStote Vnuacouaibloceta wetacedttre ates 
Linear Inverse Problems with Convex Constraints. ..........:.ccccscssectesnecsneesnsesees 
limaring with Missing Data. ici cccece ccasvsnectns ivenonsdesvecdsacnsh seas dasaleeawendnwdes 
TRVOrse SCM io05cs ceScd ce ssacudedaa adedinarauhsiisoesasasnasnceenntennne hbo tenes 
Fredholin Integral Hewat ms. 0.0... 020 csenccnesenimen samine ea baud pe antinn enna aesemeniimsacetioes 
AD St COME GOI isi steely suid dl, onier aden ete nAsetaheah ni edthcuimntettan antenna een meals ee i atesaiaN 
1 INC IOIO s caastaee hurries dati ah de aijanasanananus aes uae sume oapemeenedia eras nepamaigansuende een 


nn 


J.M. Borwein (&\) 

School of Mathematical and Physical Sciences, University of Newcastle, Newcastle, NSW, 
Australia 

e-mail: Jonathan.Borwein @newcastle.edu.au; jon.borwein@ gmail.com 


D.R. Luke 

Institute of Numerical and Applied Mathematics, Georg-August-Universitaét Gottingen, 
Gottingen, Germany 

e-mail: r.luke@math.uni-goettingen.de 


© Springer Science+Business Media New York 2015 
O. Scherzer (ed.), Handbook of Mathematical Methods in Imaging, 
DOI 10.1007/978-1-4939-0790-8_7 


258 J.M. Borwein and D.R. Luke 


MPSS RET UONOES hs clecesaosinus exaceasiiae Saasbeniead nosien eae sca ena ba encaeaee ua ee sidine eae Tsar ERS 299 
MTT Ta yrs sevoyeun dS srecetlca asere B scein ae eso a wrod nase wow rotaed on MoM aoaaaledten oe aanbaplneaturnsndeareabe 299 
Abstract 


This chapter surveys key concepts in convex duality theory and their application 
to the analysis and numerical solution of problem archetypes in imaging. 
Convex analysis, Variational analysis, Duality 


1 Introduction 


An image is worth a thousand words, the joke goes, but takes up a million times 
the memory — all the more reason to be efficient when computing with images. 
Whether one is determining a “best” image according to some criteria or applying 
a “fast” iterative algorithm for processing images, the theory of optimization and 
variational analysis lies at the heart of achieving these goals. Success or failure 
hinges on the abstract structure of the task at hand. For many practitioners, these 
details are secondary: if the picture looks good, it is good. For the specialist in 
variational analysis and optimization, however, it is what is went into constructing 
the image that matters: if it is convex, it is good. 

This chapter surveys more than a half-a-century of work in convex analysis that 
has played a fundamental role in the development of computational imaging and 
brings to light as many of the contributors to this field as possible. There is no 
shortage of good books on convex and variational analysis; interested readers are 
referred to the modern references [4, 6, 24, 27, 31,49, 61, 71, 72, 80, 93, 99, 103, 106, 
107, 110,111,118]. References focused more on imaging and signal processing, but 
with a decidedly variational flavor, include [3, 39, 109]. For general references on 
numerical optimization, see [21, 35,42, 86,97, 117]. 

For many years, the dominant distinction in applied mathematics between 
problem types has rested upon linearity, or lack thereof. This categorization still 
holds sway today with nonlinear problems largely regarded as “hard,” while linear 
problems are generally considered “easy.” But since the advent of the interior 
point revolution [96], at least in linear optimization, it is more or less agreed that 
nonconvexity, not nonlinearity, more accurately delineates hard from easy. The goal 
of this chapter is to make this case more broad. Indeed, for convex sets topological, 
algebraic, and geometric notions often coincide, and so the tools of convex analysis 
provide not only for a tremendous synthesis of ideas but also for key insights, whose 
dividends are efficient algorithms for solving large (infinite dimensional) problems, 
and indeed even large nonlinear problems. 

This chapter concerns different instances of a single optimization model. This 
model accounts for the vast majority of variational problems appearing in imaging 
science: 
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minimize J,(x) 
xeCCX (1) 
subjectto Ax e€ D. 


Here, X and Y are real Banach spaces with continuous duals X* and Y*, C and D 
are closed and convex, A: X — Y is acontinuous linear operator, and the integral 
functional I, (x) := ie g(x(t)) (dt) is defined on some vector subspace L p(T, [1) 
of X for j4, acomplete totally finite measure on some measure space 7’. The integral 
operator J, is an entropy with integrand g : R —] — 00, +00] a closed convex 
function. This provides an extremely flexible framework that specializes to most of 
the instances of interest and is general enough to extend results to non-Hilbert space 
settings. The most common examples are 


Burg entropy: g(x) := —In(x) (2) 
x In(x) (3) 
xIn(x) + (1—x)Ind-x) 4 


lll? 


II 


Shannon-—Boltzmann entropy: g(x): 


II 


Fermi—Dirac entropy : (x) : 


L, norm g(x) := —— (5) 
P 
xP x>O0 
Lp entropy g(x) := 4 ” ~ (6) 
+oo else 
Total variation g(x) := |Vx|. (7) 


See [18, 25, 26, 34, 37, 43, 44, 56, 64, 112] for these and other entropies. 

There is a rich correspondence between the algorithmic approach to applications 
implicit in the variational formulation (1) and the prevalent feasibility approach to 
problems. Here, one considers the problem of finding the point x that lies in the 
intersection of the constraint sets: 


fndxeCNS where S:={xeX |AxeD}. 


In the case where the intersection is quite large, one might wish to find the point in 
the intersection in some sense closest to a reference point x9 (frequently the origin). 
It is the job of the objective in (1) to pick the element of C M S that has the desired 
properties, that is, to pick the best approximation. The feasibility formulation 
suggests very naturally projection algorithms for finding the intersection whereby 
one applies the constraints one at a time in some fashion, e.g., cyclically, or at 
random [5, 30, 42,51,52, 119]. This is quite a powerful framework as it provides a 
great deal of flexibility and is amenable to parallelization for large-scale problems. 
Many of the algorithms for feasibility problems have counterparts for the more 
general best approximation problems [6,9, 10,58,88]. For studies of these algorithms 
in nonconvex settings, see [2,7,8, 1 1-14,29,38,53,68, 78,79, 87-89]. The projection 
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algorithms that are central to convex feasibility and best approximation problems 
play a key role in algorithms for solving the problems considered here. 

Before detailing specific applications, it is useful to state a general duality result 
for problem (1) that motivates the convex analytic approach. One of the more central 
tools is the Fenchel conjugate [62] of a mapping f : X — [-oo, +00], denoted 
f* : X* — [-00, +00] and defined by 


f*")= sup{{x", = F@)i 


The conjugate is always convex (as a supremum of affine functions), while f = 
if **|x exactly if f is convex, proper (not everywhere infinite), and lower semi- 
continuous (Isc) [24,61]. Here and below, unless otherwise specified, X is a normed 
space with dual X*. The following theorem uses constraint qualifications involving 
the concept of the core of a set, the effective domain of a function (dom /), and the 
points of continuity of a function (cont /). 


Definition 1 (Core). The core of a set F C X is defined by x € core F if for each 
he{x eX |||x|| = 1}, there exists 6 > 0 so thatx +th € F foralO <t <6. 


It is clear from the definition that int F C core F. If F is a convex subset of a 
Euclidean space, or if F' is closed, then the core and the interior are identical [27, 
Theorem 4.1.4]. 

Theorem 1 (Fenchel Duality [24, Theorems 2.3.4 and 4.4.18]). Let X and Y be 
Banach spaces, let f : X —> (—oo,+00] and g : Y — (—oo, +00] and let 


A: X -— Y bea bounded linear map. Define the primal and dual values p, 
d € [—o00, +00] by the Fenchel problems 


p= inf f (x) + g(Ax)} 


d= up {—f*(A*y*) — e*(-y*)}. (8) 
y*ey* 


Then these values satisfy the weak duality inequality p > d. 
If f, g are convex and satisfy either 


0 €core (dom g—Adom f) with f and g Isc, (9) 
or 
Adom f Mcont g 4 J, (10) 


then p = d, and the supremum to the dual problem is attained if finite. 
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Applying Theorem | to problem (1) yields f(x) = I,(x) + tc(x) and g(y) = 
tp(y) where tp is the indicator function of the set F: 


if F 
moa Vs (1) 
+oo else. 


The tools of convex analysis and the phenomenon of duality are central to 
formulating, analyzing, and solving application problems. Already apparent from 
the general application above is the necessity for a calculus of Fenchel conjugation 
in order to compute the conjugate of sums of functions. In some specific cases, one 
can arrive at the same conclusion with less theoretical overhead, but this is at the 
cost of missing out more general structures that are not necessarily automatic in 
other settings. 

Duality has a long-established place in economics where primal and dual 
problems have direct interpretations in the context of the theory of zero-sum games, 
or where Lagrange multipliers and dual variables are understood, for instance, as 
shadow prices. In imaging, there is not as often an easy interplay between the 
physical interpretation of primal and dual problems. Duality has been used toward a 
variety of ends in contemporary image and signal processing, the majority of them, 
however, having to do with algorithms [17, 33, 34, 43-46, 54, 55,57, 70, 73, 90, 116]. 
Nevertheless, the dual perspective yields new statistical or information theoretic 
insight into image processing problems, in addition to faster algorithms. Since the 
publication of the first edition of this handbook, interest in convex duality theory 
has only continued to grow. This is, in part, due to the now ubiquitous application of 
convex duality techniques to nonconvex problems; as heuristics, through appropriate 
convex relaxations, or otherwise [48, 114, 115]. As a measure of this growing 
interest, a Web of Science search for articles published between 2011 and 2013 
having either “convex relaxation” or “primal dual” in the title, abstract, or keywords, 
returns a combined count of approximately 1,500 articles. 

Modern optimization packages heavily exploit duality and convex analysis. A 
new trend that has matured in recent years is the field of computational convex 
analysis which employs symbolic, numerical, and hybrid computations of objects 
like the Fenchel conjugate [23, 65, 81-85, 91,98]. Software suites that rely heavily 
on a convex duality approach include: CCA (Computational Convex Analysis, http:// 
atoms.scilab.org/toolboxes/CCA) for Scilab, CVX and its extensions (http://cvxr. 
com/cvx/) for MATLAB (including a C code generator) [65, 91], and S-CAT 
(Symbolic Convex Analysis Toolkit, http://web.cs.dal.ca/~chamilto/research.html) 
for Maple [23]. For a review of the computational aspects of convex analysis see 
[84]. 

In this chapter, five main applications illustrate the variational analytical 
approach to problem solving: linear inverse problems with convex constraints, 
compressive imaging, image denoising and deconvolution, nonlinear inverse 
scattering, and finally Fredholm integral equations. A brief review of these 
applications is presented below. Subsequent sections develop the tools for their 
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analysis. At the end of the chapter these applications are revisited in light of the 
convex analytical tools collected along the way. 

The application of compressive sensing, or more generally sparsity optimization 
leads to a new theme that is emerging out of the theory of convex relaxations 
for nonconvex problems, namely direct global nonconvex methods for structured 
nonconvex optimization problems. Starting with the seminal papers of Candés 
and Tao [40,41], the theory of convex relaxation for finding the sparsest vector 
satisfying an underdetermined affine constraint has concentrated on determining 
sufficient conditions under which the solution to a relaxation of the problem 
to £; minimization with an affine equality constraint corresponds exactly to the 
global solution of the original nonconvex sparsity optimization problem. These 
conditions have been used in recent years to guarantee global convergence of simple 
projected gradient- and alternating projections-type algorithms for solving simpler 
nonconvex optimization problems whose global solutions correspond to a solution 
of the original problem (see [15, 16, 19, 20, 63, 69] and references therein). This 
is the natural point at which this chapter leaves off, and the frontier of nonconvex 
programming begins. 


Linear Inverse Problems with Convex Constraints 


Let X be a Hilbert space and g(x) := $|[x \|?. The integral functional /, is the usual 
L> norm and the solution is the closest feasible point to the origin: 


minimize ; Il ||? 
xeCCX (12) 
subject to Ax =b. 


Levi, for instance, used this variational formulation to determine the minimum 
energy band-limited signal that matched N measurements b € R"” with the model 
A: X — R" [77]. Note that the signal space is infinite dimensional while the 
measurement space is finite dimensional, a common situation in practice. Potter and 
Arun [100] recognized a much broader applicability of this variational formulation 
to remote sensing and medical imaging and applied duality theory to characterize 
solutions to (12) by X = Pc A*(¥), where y € Y satisfies b = APcA*y [100, 
Theorem 1]. Particularly attractive is the feature that when Y is finite dimensional, 
these formulas yield a finite dimensional approach to an infinite dimensional 
problem. The numerical algorithm suggested by Potter and Arun is an iterative 
procedure in the dual variables: 


Viti = yj + v¥G@—APcA*y;) j =0,1,2,... (13) 


The optimality condition and numerical algorithms are explored at the end of this 
chapter. 
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As satisfying as this theory is, there is a crucial assumption in the theorem of 
Potter and Arun about the existence of y € Y such that b = APc A*y; one need 
only to consider linear least squares, for an example, where the primal problem is 
well posed, but no such y exists [22]. A specialization of Theorem | to the case of 
linear constraints facilitates the argument. The next corollary is a specialization of 
Theorem 1, where g is the indicator function of the point b in the linear constraint. 


Corollary 1 (Fenchel Duality for Linear Constraints). Given any f : X > 
(—oo, co], any bounded linear map A: X — Y, and any element b € Y, the 
following weak duality inequality holds: 


ing (/() Ax = b}> sup {(b. y*) — f*A"y 9h 


If f is lsc and convex and b € core(Adom f), then equality holds and the 
supremum is attained if finite. 


Suppose that C = X, a Hilbert space and A: X — X. The Fenchel dual to 
(12) is 


1 
maximize (y, b) — || A*y||’. (14) 
yex 2 


(The L»2 norm is self-dual.) Suppose that the primal problem (12) is feasible, that 
is, b € range(A). The objective in (14) is convex and differentiable, so elementary 
calculus (Fermat’s rule) yields the optimal solution y with AA*y = b, assuming y 
exists. If the range of A is strictly larger than that of 4 A*, however, it is possible 
to have b € range(A) but b ¢ range(AA*), in which case the optimal solution ¥ 
to (12) is not equal to A*Y, since y is not attained. For a concrete example see [22, 
Example 2.1]. 


Imaging with Missing Data 


Let X = R” and g(x) := ||x||, for p = 0 or p = 1. The case p = 1 is the £; 
norm, and by ||x||o is meant the function 


lIxllo:= }> | sign(x,)I, 
i 


where sign(0) := 0. This yields the optimization problem 


minimize |x| p 
xeR" (15) 
subject to Ax =b. 
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This model has received a great deal of attention recently in applications of 
compressive sensing where the number of measurements is much smaller than the 
dimension of the signal space, that is, b € R” form <«_ n. This problem is well 
known in statistics as the missing data problem. 

For €; optimization (p = 1), the seminal work of Candés and Tao establishes 
probabilistic criteria for when the solution to (15) is unique and exactly matches 
the true signal x, [41]. Sparsity of the original signal x, and the algebraic structure 
of the matrix A are key requirements. Convex analysis easily yields a geometric 
interpretation of these facts. The dual to this problem is the linear program 


maximize bly 
yeR™ (16) 
subject to (A™y); € Fld) 7 = h2sesam: 


Deriving this dual is one of the goals of this chapter. Elementary facts from 
linear programming guarantee that the solution includes a vertex of the polyhedron 
described by the constraints, and hence, assuming A is full rank, there can be at most 
m active constraints. The number of active constraints in the dual problem provides 
an upper bound on the number of nonzero elements in the primal variable — the 
signal to be recovered. Unless the number of nonzero elements of x, is less than 
the number of measurements m, there is no hope of uniquely recovering x.. The 
uniqueness of solutions to the primal problem is easily understood in terms of the 
geometry of the dual problem, that is, whether or not solutions to the dual problem 
reside along the edges or faces of the polyhedron. More refined details about how 
sparse x» needs to be in order to have a reasonable hope of exact recovery require 
more work, but elementary convex analysis already provides the essential intuition. 

For the function ||x|}o (p = O in (15)) the equivalence of the primal and dual 
problems is lost due to the nonconvexity of the objective. The theory of Fenchel 
duality still yields weak duality, but this is of limited use in this instance. The 
Fenchel dual to (15) is 


maximize bly 
yeR” (17) 
subject to (A*y); =0 fF =1,2,..... 


Denoting the values of the primal (15) and dual problems (17) by p and d, 
respectively, these values satisfy the weak duality inequality p > d. The primal 
problem is a combinatorial optimization problem, and hence NP-hard; the dual 
problem, however, is a linear program, which is finitely terminating. Relatively 
elementary variational analysis provides a lower bound on the sparsity of signals 
x that satisfy the measurements. In this instance, however, the lower bound only 
reconfirms what is already known. Indeed, if A is full rank, then the only solution 
to the dual problem is y = 0. In other words, the minimal sparsity of the solution 
to the primal problem is zero, which is obvious. The loss of information in passing 
from primal to dual formulations of nonconvex problems is a common phenomenon 
and underscores the importance of convexity. 
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The Fenchel conjugates of the €; norm and the function ||-||9 are given 
respectively by 
« 0 |lylleo <1 
91 (Y) = | . (gi (x) = [Ix|h) (18) 
+oo else 
, 0 y=0 
Yo (¥) *= | (Yo(x) *= |Ixllo) (19) 
+oo else 


/p 
It is not uncommon to consider the function || - ||o as the limit of (= (eal? ) as 


p — O. This suggests an alternative approach based on the regularization of the 
conjugates. For L and e > 0 define 


Z (eee elim) = ae) (y €[-L, L)) 


2L In(2) In(2) 


Ge (V) = (20) 


+00 for |y| > L. 


This is a scaled and shifted Fermi—Dirac entropy (4). It is also a smooth convex 
function on the interior of its domain and so elementary calculus can be used to 
calculate the Fenchel conjugate, 


€ 
In(2) 


gr (x) = In (474/41) —xL—e. (21) 
For L > 0 fixed, 


0 y €[-L,L] 


lim g..1(y) = and lim g?,, (x) = L|x|. 
e>0 +oo else e>0 °° 
For € > 0 fixed 
r=0 at 
1 = d 1 := 0. 
Jim ge,1 ) is ae and lim 9.1 (x) = 0 
Note that ||-|lo and g*) := 0 have the same conjugate, but unlike ||-|lo the 


biconjugate of y*y is itself. Also note that gy, and y*, are convex and smooth 
on the interior of their domains for all ¢, L > 0. This is in contrast to metrics of the 


form (>) |x; 1’) which are nonconvex for p < 1. This suggests solving 


minimize Ty* (x) 
xeR" 6,L (22) 
subject to Ax =b 
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as a smooth convex relaxation of the conventional £, optimization for 0 < p < 1. 
For further details see [28]. 


Image Denoising and Deconvolution 


Consider next problems of the form 
inimize I,(x) + : || A y|P (23) 
munimize g(x x 


where X is a Hilbert space, Jy : X — (—oo,+00] is a semi-norm on X, and 
A: X — Y, is a bounded linear operator. This problem is explored in [17] 
as a general framework that includes total variation minimization [108], wavelet 
shrinkage [59], and basis pursuit [47]. When A is the identity, problem (23) amounts 
to a technique for denoising; here y is the received, noisy signal, and the solution 
xX iS an approximation with the desired statistical properties promoted by the 
objective J,. When the linear mapping A is not the identity (for instance, A models 
convolution against the point spread function of an optical system) problem (23) is 
a variational formulation of deconvolution, that is, recovering the true signal from 
the image y. The focus here is on total variation minimization. 

Total variation was first introduced by Rudin et al. [108] as a regularization 
technique for denoising images while preserving edges and, more precisely, the 
statistics of the noisy image. The total variation of an image x € X = L,(T)- 
for T and open subset of R? — is defined by 


Try (x) := sup} [x00 div &(t)dt |é € C)(T,R), |EO| <1 Vte TS. 


The integral functional /;y is finite if and only if the distributional derivative Dx of 
x is a finite Radon measure in 7, in which case [ry (x) = |Dx|(T). If, moreover, 
x has a gradient Vx € L,(T,R7), then [py (x) = i |Vx(t)|dt, or, in the context 
of the general framework established at the beginning of this chapter, Jry(x) = 
I(x) where g(x(t)) := |Vx(t)|. The goal of the original total variation denoising 
problem proposed in [108] is then to 


oh I 
minimize ty (x) on 
subject to J, Ax = f-x0 and f,|Ax —x0|? = 07. 


The first constraint corresponds to the assumption that the noise has zero mean 
and the second assumption requires the denoised image to have a predetermined 
standard deviation o. Under reasonable assumptions [44], this problem is equivalent 
to the convex optimization problem 
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minimize Try (x) (25) 
subject to || Ax — xol|? < 0. 

Several authors have exploited duality in total variation minimization for efficient 
algorithms to solve the above problem [43, 46, 57, 70]. One can “compute” the 
Fenchel conjugate of [ry indirectly by using the already mentioned property that the 
biconjugate of a proper, convex Isc function is the function itself: f**(x) = f(x) 
if (and only if) f is proper, convex, and Isc at x. Rewriting Jy as the Fenchel 
conjugate of some function yields 


Irv (x) = sup (x, v) —tx(v), 


where 


K := {divé |[E€CI(.R2) and [&()| <1 Wie Th. 


From this, it is then clear that the Fenchel conjugate of Jy is the indicator function 
of the convex set K, tx. 

In [43], duality is used to develop an algorithm, with proof of convergence, for 
the problem 


ei 1 
minimize Irv (x) + >llx — oll? (26) 


with X a Hilbert space. First-order optimality conditions for this unconstrained 
problem are 


O0E€x—-—x +AdIry(x), (27) 
where d/ry (x) is the subdifferential of Iypy at x defined by 
vedlry(x) = Irv(Qy) = Irv(x)+ (vy, y-x)) Vy. 
The optimality condition (27) is equivalent to [24, Proposition 4.4.5] 


x € Oty ((xo — x)/A) (28) 
or, since [7 = lk, 

Wel pa oe VG 

rt ee 


where z = (Xo — x)/A. (For the finite dimensional statement, see [71, Proposition 
1.6.1.2].) Since K is convex, standard facts from convex analysis determine that 
0tx (z) is the normal cone mapping to K at z, denoted Nx (z) and defined by 
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{ve X |(vy,x-—z) <0 forallxe K} zek 


Nx(z):= tek. 


Note that this is a set-valued mapping. The resolvent (I + Ou x) evaluated at 
xo/A is the orthogonal projection of x9/A onto K. That is, the solution to (26) is 


Xx = Xo — Px (x0/A) = x0 — Pax (x0). 


The inclusions disappear from the formulation due to convexity of K: the resolvent 
of the normal cone mapping of a convex set is single valued. The numerical algo- 
rithm for solving (26) then amounts to an algorithm for computing the projection 
Pyx. The tools from convex analysis used in this derivation are the subject of this 
chapter. 


Inverse Scattering 


An important problem in applications involving scattering is the determination of 
the shape and location of scatterers from measurements of the scattered field at a 
distance. Modern techniques for solving this problem use indicator functions to 
detect the inconsistency or insolubility of an Fredholm integral equation of the 
first kind, parameterized by points in space. The shape and location of the object 
is determined by those points where the auxiliary problem is solvable. Equivalently, 
the technique determines the shape and location of the scatterer by determining 
whether a sampling function, parameterized by points in space, is in the range of a 
compact linear operator constructed from the scattering data. 

These methods have enjoyed great practical success since their introduction 
in the latter half of the 1990s. Recently Kirsch and Grinberg [74] established a 
variational interpretation of these ideas. They observe that the range of a linear 
operator G : X — Y (X and Y are reflexive Banach spaces) can be characterized 
by the infimum of the mapping 


hi): Y* > RU {-c0, +00} := |(W F)I, 


where F := GSG* for S : X* — X, a coercive bounded linear operator. 
Specifically, they establish the following. 


Theorem 2 ([74, Theorem 1.16]). Let X,Y be reflexive Banach spaces with duals 
X* and Y*. Let F: Y* > Y andG: X —> Y be bounded linear operators with 
F = GSG* for S : X* — X a bounded linear operator satisfying the coercivity 
condition 


(vy, Sv)| > cllelly* for some c > 0 and all g € range(G*) C X*. 
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Then for any @ € Y \{0} @ € range(G) if and only if 


infth(W) |W e Y*(G, W) = 1p >0. 


It is shown below that the infimal characterization above is equivalent to the 
computation of the effective domain of the Fenchel conjugate of h, 


n°) = sup (6. ¥) —hO)} (29) 


In the case of scattering, the operator F above is an integral operator whose 
kernel is made up of the “measured” field on a surface surrounding the scatterer. 
When the measurement surface is a sphere at infinity, the corresponding operator 
is known as the far field operator. The factor G maps the boundary condition of 
the governing PDE (the Helmholtz equation) to the far field pattern, that is, the 
kernel of the far field operator. Given the right choice of spaces, the mapping G 
is compact, one-to-one, and dense. There are two keys to using the above facts for 
determining the shape and location of scatterers: first, the construction of the test 
function ¢@ and, second, the connection of the range of G to that of some operator 
easily computed from the far field operator F. The secret behind the success of 
these methods in inverse scattering is, first, that the construction of ¢ is trivial and, 
second, that there is (usually) a simpler object to work with than the infimum in 
Theorem 2 that depends only on the far field operator (usually the only thing that 
is known). Indeed, the test functions ¢ are simply far field patterns due to point 
sources: @, := e~'***<, where % is a point on the unit sphere (the direction of the 
incident field), k is a nonnegative integer (the wave number of the incident field), 
and z is some point in space. 

The crucial observation of Kirsch is that @, is in the range of G if and only if z 
is a point inside the scatterer. If one does not know where the scatter is, let alone 
its shape, then one does not know G, however, the Fenchel conjugate depends not 
on G but on the operator F which is constructed from measured data. In general, 
the Fenchel conjugate, and hence the Kirsch—Grinberg infimal characterization, is 
difficult to compute, but depending on the physical setting, there is a functional U of 
F under which the ranges of U(F) and G coincide. In the case where F is a normal 
operator, U(F) = (F*F)!/*; for non-normal F, the functional U depends more 
delicately on the physical problem at hand and is only known in a handful of cases. 
So the algorithm for determining the shape and location of a scatterer amounts to 
determining those points z, where e~‘***< js in the range of U(F’) and where U and 
F are known and easily computed. 


Fredholm Integral Equations 


In the scattering application of the previous section, the prevailing numerical 
technique is not to calculate the Fenchel conjugate of (yw) but rather to explore 
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the range of some functional of F’. Ultimately, the computation involves solving a 
Fredholm integral equation of the first kind, returning to the more general setting 
with which this chapter began. Let 


(Ax)(s) = i a(s,t)u(dt) = b(s) 


for reasonable kernels and operators. If A is compact, for instance, as in most 
deconvolution problems of interest, the problem is ill posed in the sense of 
Hadamard. Some sort of regularization technique is therefore required for numerical 
solutions [60, 66,67, 76, 113]. Regularization is explored in relation to the constraint 
qualifications (9) or (10). 

Formulating the integral equation as an entropy minimization problem yields 


minimize Ig(x) 
xeX (30) 
subject to Ax =b. 


Following [22, Example 2.2], let T and S be the interval [0,1] with Lebesgue 
measures jz and v, and let a(s,t) be a continuous kernel of the Fredholm operator 
A mapping X := C({0,1]) to Y := C(O, 1]), both equipped with the supremum 
norm. The adjoint operator is given by A*y = {J a(s, t)A(ds)} jL(dt), where the 
dual spaces are the spaces of Borel measures, X¥* = M((0, 1]) and Y* = M((0, 1)). 
Every element of the range is therefore jz-absolutely continuous and A* can be 
viewed as having its range in L;([0, 1], jz). It follows from [105] that the Fenchel 
dual of (30) for the operator A is therefore 


max (b, y*) — Igx(A*y*). (31) 


y*ey* 


Note that the dual problem, unlike the primal, is unconstrained. Suppose that A 
is injective and that b € range(A). Assume also that y* is everywhere finite and 
differentiable. Assuming the solution y to the dual is attained, the naive application 
of calculus provides that 


b=A (Fun) and xy = (= (a*y)). (32) 
or or 

Similar to the counterexample explored in section “Linear Inverse Problems 
with Convex Constraints”, it is quite likely that A(2& (range(A*))) is smaller 
than the range of A, hence it is possible to have b € range(A) but not in 
A (2 (range(4*))). Thus the assumption that the solution to the dual problem 

is attained cannot hold and the primal—dual relationship is broken. 
For a specific example, following [22, Example 2.2], consider the Laplace 
transform restricted to [0, 1]: a(s,t) := e~* (s € [0,1]), and let g be either the 
Boltzmann-Shannon entropy, Fermi—Dirac entropy, or an L, norm with p ¢€ (1,2), 
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(3)-(5), respectively. Take b(s) := Soo. e-'x(t)dt for ¥ := a |t - 5| + Ba 
solution to (30). It can be shown that the restricted Laplace operator defines an 
injective linear operator from C((0, 1]) to C([0, 1]). However, x, given by (32) is 
continuously differentiable and thus cannot match the known solution x which is not 
differentiable. Indeed, in the case of the Boltzmann—Shannon entropy, the conjugate 
function and A*y are entire hence the ostensible solution x, must be infinitely 
differentiable on [0, 1]. One could guarantee that the solution to the primal problem 
(30) is attained by replacing C((0, 1]) with L,({0, 1]), but this does not resolve the 
problem of attainment in the dual problem. 

To recapture the correspondence between primal and dual problems it is neces- 
sary to regularize or, alternatively, relax the problem, or to require the constraint 
qualification b € core(Adom ¢). Such conditions usually require A to be 
surjective, or at least to have closed range. 


2 Background 


As this is meant to be a survey of some of the more useful milestones in convex 
analysis, the focus is more on the connections between ideas than their proofs. The 
reader will find the proofs in a variety of sources. The presentation is by default in 
a normed space X with dual X*, though if statements become too technical, the 
Euclidean space variants will suffice. E denotes a finite-dimensional real vector 
space R” for some n € N endowed with the usual norm. Typically, X will be 
reserved for a real infinite-dimensional Banach space. A common convention in 
convex analysis is to include one or both of —oo and +00 in the range of functions 
(typically only +00). This is denoted by the (semi-) closed interval (—oo, +00] or 
[—co, +o0]. 

A set C C X is said to be convex if it contains all line segments between any 
two points in C: Ax + (1—A)y e€ C for all A € [0,1] and x,y € C. Much of 
the theory of convexity is centered on the analysis of convex sets, however, sets and 
functions are treated interchangeably through the use of level sets, epigraphs, and 
indicator functions. The lower-level sets of a function f : X — [—oo, +00] are 
denoted lev <, f and defined by levy f := {x € X | f(x) < a} wherea € R. The 
epigraph of a function f : X — [—oo, +00] is defined by 


epi f := {(x,t) € ExR| f(x) <t}. 


This leads to the very natural definition of a convex function as one whose epigraph 
is a convex set. More directly, a convex function is defined as a mapping f : X > 
[—o0, +00] with convex domain and 


fAx+(U—-A)y) <Af(x)+—-A) f(y) for any x, y € dom f and 4d € (0, 1]. 


A proper convex function f : X — [—oo, +00] is strictly convex if the above 
inequality is strict for all distinct x and y in the domain of f and all 0 <A < 1. 
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A function is said to be closed if its epigraph is closed; whereas a lower semi- 
continuous (Isc) function f satisfies liminf,++ f(x) => f(%) for all x € X. These 
properties are in fact equivalent: 


Proposition 1. The following properties of a function f : X — [—co, +00] are 
equivalent: 


(i) f is Isc. 
(ii) epi f is closed in X xR. 
(iii) The level sets lev <q f are closed on X for eacha € R. 


Guide. For Euclidean spaces, this is shown in [107, Theorem 1.6]. In the Banach 
space setting this is [24, Proposition 4.1.1]. This is left as an exercise for the Hilbert 
space setting in [50, Exercise 2.1]. a 


A principal focus is on proper functions, that is, f : E — [—oo, +00] with 
nonempty domain. The indicator function is often used to pass from sets to functions 


0 xEC 


ic(x) := 
+oo else. 


For C c X convex, f : C — [—co, +00] will be referred to as a convex function 
if the extended function 


f(x) xEC 


+oo else 


is convex. 


Lipschitzian Properties 


Convex functions have the remarkable, yet elementary, property that local bound- 
edness and local Lipschitz properties are equivalent without any additional assump- 
tions on the function. In the following statement of this fact, the unit ball is denoted 
by By := {x € X | ||x|| < 1}. 


Lemma 1. Let f : X — (—co,+00] be a convex function and suppose that 
C C X is a bounded convex set. If f is bounded on C + 5By for some 6 > 0, then 
f is Lipschitz on C. 

Guide. See [24, Lemma 4.1.3]. | 


With this fact, one can easily establish the following. 
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Proposition 2 (Convexity and Continuity in Normed Spaces). Let f : X — 
(—oco, +00] be proper and convex, and let x € dom f. The following are 
equivalent: 


(i) f is Lipschitz on some neighborhood of x. 
(ii) f is continuous at x. 
(iii) f is bounded on a neighborhood of x. 
(iv) f is bounded above on a neighborhood of x. 


Guide. See [24, Proposition 4.1.4] or [31, Sect. 4.1.2]. | 
In finite dimensions, convexity and continuity are much more tightly connected. 


Proposition 3 (Convexity and Continuity in Euclidean Spaces). Let f : E > 
(—oo, +00] be convex. Then f is locally Lipschitz, and hence continuous, on the 
interior of its domain. 


Guide. See [24, Theorem 2.1.12] or [72, Theorem 3.1.2] | 


Unlike finite dimensions, in infinite dimensions a convex function need not be 
continuous. A Hamel basis, for instance, an algebraic basis for the vector space 
can be used to define discontinuous linear functionals [24, Exercise 4.1.21]. For 
Isc convex functions, however, the correspondence follows through. The following 
statement uses the notion of the core of a set given by Definition 1. 


Example I (A Discontinuous Linear Functional). Let coo denote the normed sub- 
space of all finitely supported sequences in co, the vector space of sequences in X 
converging to 0; obviously coo is open. Define A : cop > R by A(x) = Ox; 
where x = (x;) € Cog. This is clearly a linear functional and discontinuous at 0. 
Now extend A to a functional A on the Banach space co by taking a basis for co 
considered as a vector space over Coo. In particular, C := A7!([-1, 1]) is a convex 
set with empty interior for which 0 is a core point. Moreover, C = co and A is 
certainly discontinuous. | 


Proposition 4 (Convexity and Continuity in Banach Spaces). Suppose X is a 
Banach space and f : X — (—co, +00] is Isc, proper, and convex. Then the 
following are equivalent: 


(i) f is continuous at x. 
(ii) x € int dom f. 
(iii) x € core dom f. 


Guide. This is [24, Theorem 4.1.5]. See also [31, Theorem 4.1.3]. x 


The above result is helpful since it is often easier to verify that a point is in the core 
of the domain of a convex function than in the interior. 


274 J.M. Borwein and D.R. Luke 


Subdifferentials 


The analog to the linear function in classical analysis is the sublinear function in 
convex analysis. A function f : X — [—oo, +00] is said to be sublinear if 


fAx +yy) <Af(x) + vf) forallx,y € X andda,y > 0. 


By convention, 0-(+00) = 0. Sometimes sublinearity is defined as a function f 
that is positively homogeneous (of degree 1) —i.e.,0 € dom f and f(Ax) = Af(x) 
for all x and all A > 0 —and is subadditive 


f(x +y) < f(x) + f(y) — for all x and y. 


Example 2 (Norms). A norm on a vector space is a sublinear function. Recall that 
a nonnegative function || - || on a vector space X is a norm if 


(i) ||x|| => O for eachx € X. 

(ii) ||x|| = 0 if and only if x = 0. 
(iii) ||Ax|| = |A||x|| for every x € X and scalar A. 
(iv) [lx + yll < Ilxll + lly] for every x, y € X. 


A normed space is a vector space endowed with such a norm and is called a Banach 
space if it is complete which is to say that all Cauchy sequences converge. a 


Another important sublinear function is the directional derivative of the function 
f at x in the direction d defined by 


f@ +td) — f(*) 


t 


; ; 
x:d):= lim 
f(x; d) ne 

whenever this limit exists. 


Proposition 5 (Sublinearity of the Directional Derivative). Let X be a Banach 
space and let f : X — (—o0,+00] be a convex function. Suppose that X € 
core(dom f). Then the directional derivative f'(X;-) is everywhere finite and 
sublinear. 


Guide. See [31, Proposition 4.2.4]. For the finite dimensional analog, see [72, 
Proposition D.1.1.2] or [24, Proposition 2.1.17]. | 


Another important instance of sublinear functions are support functions of 
convex sets, which, in turn permit local first-order approximations to convex 
functions. A support function of a nonempty subset S of the dual space X*, usually 
denoted os, is defined by os(x) := sup {(s, x) |s € S}. The support function is 
convex, proper (not everywhere infinite), and 0 € dom os. 
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Example 3 (Support Functions and Fenchel Conjugation). From the definition of 
the support function it follows immediately that, for a closed convex set C, 


* 2K 
lc =Oc and tc =lc¢. a 


A powerful observation is that any closed sublinear function can be viewed as a 
support function. This can be seen by the representation of closed convex functions 
via affine minorants. This is the content of the Hahn—Banach theorem, which is 
stated in infinite dimensions as this setting will be needed below. 


Theorem 3 (Hahn-Banach: Analytic Form). Let X be a normed space and o : 
X — R be acontinuous sublinear function with dom o = X. Suppose that L is a 
linear subspace of X and that the linear function h : L — R is dominated by o on 
L, that is o => hon L. Then there is a linear function minorizing o on X, that is, 
there exists a x* € X* dominated by o such that h(x) = (x*, x) < o(x) for all 
xeL. 


Guide. The proof can be carried out in finite dimensions with elementary tools, 
constructing x* from h sequentially by one-dimensional extensions from L. See 
[72, Theorem C.3.1.1] and [24, Proposition 2.1.18]. The technique can be extended 
to Banach spaces using Zorn’s lemma and a verification that the linear functionals so 
constructed are continuous (guaranteed by the domination property) [24, Theorem 
4.1.7]. See also [110, Theorem 1.11]. a 


An important point in the Hahn—Banach extension theorem is the existence of a 
minorizing linear function, and hence the existence of the set of linear minorants. 
In fact, o is the supremum of the linear functions minorizing it. In other words, o is 
the support function of the nonempty set 


So = {s € X* | (s, x) <o(x) forallx eX}. 


A number of facts follow from Theorem 3, in particular the nonemptiness of the 
subdifferential, a sandwich theorem and, thence, Fenchel Duality (respectively, 
Theorems 5, 7, and 12). It turns out that the converse also holds, and thus these facts 
are actually equivalent to nonemptiness of the subdifferential. This is the so-called 
Hahn—Banach/Fenchel duality circle. 

As stated in Proposition 5, the directional derivative is everywhere finite and 
sublinear for a convex function f at points in the core of its domain. In light of the 
Hahn-Banach theorem, then f’(X, -) can be expressed for all d € X in terms of its 
minorizing function: 


f'(%,d) = os(d) = max{(v, d)}. 


The set S for which /’(x, d) is the support function has a special name: the subd- 
ifferential of f at X. It is tempting to define the subdifferential this way, however, 
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there is a more elemental definition that does not rely on directional derivatives 
or support functions, or indeed even the convexity of f/f. The correspondence 
between directional derivatives of convex functions and the subdifferential below 
is a consequence of the Hahn—Banach theorem. 


Definition 2 (Subdifferential). For a function f : X — (—oo, +00] and a point 
x € dom f, the subdifferential of f at x, denoted df (x) is defined by 


Of (X) = {v € X* | v(x) — v(X) < f(x) — f(X) for allx € X}. 
When x ¢ dom f/f, define df (x) = ©. 
In Euclidean space the subdifferential is just 

Of (x) = {ve E |(v, x) — (vy, x) < f(x) — fC) for all x € FE}. 
An element of 0f(x) is called a subgradient of f at x. See (31,93, 107] for more in- 
depth discussion of the regular, or limiting subdifferential defined here, in addition 
to other useful varieties. This is a generalization of the classical gradient. Just as the 
gradient need not exist, the subdifferential of a lsc convex function may be empty at 
some points in its domain. Take, for example, f(x) = —V1— x? for—1 <x <1. 
Then 0f(x) = Oforx = +1. 
Example 4 (Common Subdifferentials). 


(i) Gradients. A function f : X — R is said to be strictly differentiable at X if 


tim LV LM-VI@O= 0) _ 

im =0. 

XX UX || x = ul| 
This is a stronger differentiability property than Fréchet differentiability since 
it requires uniformity in pairs of points converging to x. Luckily for convex 
functions the two notions agree. If f is convex and strictly differentiable 
at x, then the subdifferential is exactly the gradient. (This follows from 
the equivalence of the subdifferential in Definition 2 and the basic limiting 
subdifferential defined in [93, Definition 1.77] for convex functions and [93, 
Corollary 1.82].) In finite dimensions, at a point x € dom f for f convex, 
Fréchet and Gateaux differentiability coincide, and the subdifferential is a 
singleton [24, Theorem 2.2.1]. In infinite dimensions, a convex function f that 
is continuous at X is Gateaux differentiable at x if and only if the df(X) is a 
singleton [24, Corollary 4.2.11]. 

(ii) The subdifferential of the indicator function. 


dic (X) = Nc), 
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where C C X is closed and convex, X is a Banach, and Nc (x) C X™* is the 
normal cone mapping to C at x defined by 


{ve X* |(v, x -x) <0 forallxeC} xXEC 


Nc (x) := r¢ é 


(33) 


See (41) for alternative definitions and further discussion of this important 


mapping. 
(iii) Absolute value. For x € R, 


—l x <0 
d|-|(*) = 4[-1,1] x=0 
1 x >0. 


The following elementary observation suggests the fundamental significance of 
subdifferential in optimization. 


Theorem 4 (Subdifferential at Optimality: Fermat’s Rule). Let X be anormed 
space, and let f : X — (—oo, +00] be proper and convex. Then f has a (global) 
minimum at X if and only if 0 € Of (x). 


Guide. The first implication of the global result follows from a more general local 
result [93, Proposition 1.114] by convexity; the converse statement follows from the 
definition of the subdifferential and convexity. a 


Returning now to the correspondence between the subdifferential and the 
directional derivative of a convex function f’(x; d) has the following fundamental 
result. 


Theorem 5 (Max Formula — Existence of df). Let X be a normed space, d € X 
and let f : X — (—oo, +00] be convex. Suppose that x € cont f. Then df (x) # 
OD and 


St '(&%, d) = max {(x*, d) |x* € af(x)}. 


Proof. The tools are in place for a simple proof that synthesizes many of the 
facts tabulated so far. By Proposition 5 f’(X;-) is finite; so, for fixed d € 
{x € X | |x|] = 1}, let w = f’(%;d) < oo. The stronger assumption that ¥ € 
cont f and the convexity of f’(x; -) yield that the directional derivative is Lipschitz 
continuous with constant K. Let S := {td |t € R} and define the linear function 
A: S > R by A(td) := ta fort € R. Then A(-) < f’(x;-) on S. The 
Hahn-Banach theorem 3 then guarantees the existence of ¢ € X* such that 
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¢g=AonS, ¢(-)< f'(%;-)onX. 
Then ¢ € df (xX) and d(sd) = f'(X; sd) forall s > 0. (.] 


A simple example on R illustrates the importance of the qualification x € 
cont f. Let 


—/x, x>0 
f(x): R= (—co, +00] := Sa — 
+00 otherwise. 
For this example, 0f(0) = @. 
An important application of the Max formula in finite dimensions is the mean 
value theorem for convex functions. 


Theorem 6 (Convex Mean Value Theorem). Let f : E — (—oo,+0c0] be 
convex and continuous. For u,v € E there exists a point z € E interior to the 
line segment |u, v| with 


f@ — fv) < (wv, u-v) forall w € of(z). 


Guide. See [93, 107] for extensions of this result and detailed historical back- 
ground. a 


The next theorem is a key tool in developing a subdifferential calculus. It relies 
on assumptions that are used frequently enough that it is worthwhile to present them 
separately. 


Assumption 6. Let X and Y be Banach spaces and let T : X — Y be a bounded 
linear mapping. Let f : X — (—oo, +00] and g : Y > (—oo, +00] satisfy one 


of 
0 € core(dom g — T dom f) and both f and g are Isc, (34) 


or 
T dom f Ncont g 4 JZ. (35) 


The later assumption can be used in incomplete normed spaces as well. 


Theorem 7 (Sandwich Theorem). Let X and Y be Banach spaces and let T : 
X — Y bea bounded linear mapping. Suppose that f : X — (—oo, +00] and 
g: Y — (—oo, +00] are proper convex functions with f > —g oT and which 
satisfy Assumption 6. Then there is an affine function A : X —> R defined by 
Ax := (T*y*, x) +r satisfying f > A > —g oT. Moreover, for any X satisfying 
I (X) = (-g 0 T)(X), it holds that —y* € dg(TX). 
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Success Failure 


Fig. 1 Hahn—Banach sandwich theorem and its failure. (a) Success; (b) failure 


Guide. By the development to this point, the Max formula [24, Theorem 4.1.18] 
would apply to prove the result. For a vector space version see [110, Corollary 
2.1]. Another route is via Fenchel duality which is explored in the next section. 
A third approach closely related to the Fenchel duality approach [31, Theorem 
4.3.2] is based on a decoupling lemma which is also presented in the next section 
(Lemma 3). a 


Corollary 2 (Basic Separation). Let C C X be a nonempty convex set with 
nonempty interior in a normed space, and suppose Xo ¢ intC. Then there exists 


gb € X*\{0} such that 


supd@ < (xo) and (x) < (xo) forall x € intC. 
(e; 


If xo ¢ C then it can be assumed that supc @ < (Xo). 


Proof. Assume without loss of generality that 0 € int C and apply the sand- 
wich theorem with f = 4,3, T the identity mapping on X and g(x) = 
inf {r > 0 |x erC}—1. See [31, Theorem 4.3.8] and [24, Corollary 4.1.15]. 


The Hahn—Banach theorem 3 can be seen as an easy consequence of the sandwich 
theorem 7, which completes part of the circle. Figure | illustrates these ideas. 

In the next section Fenchel duality is added to this cycle. A calculus of 
subdifferentials and a few fundamental results connecting the subdifferential to 
classical derivatives and monotone operators conclude the present section. 


Theorem 8 (Subdifferential Sum Rule). Let X and Y be Banach spaces, T : 
X — Y a bounded linear mapping and let f : X — (—o0o,+00] andg: Y > 
(—oo, +00] be convex functions. Then at any point x € X 


d(f + oT) (x) Dd af(x) + T*(0g(Tx)), 


with equality if Assumption 6 holds. 


280 J.M. Borwein and D.R. Luke 


Proof sketch. The inclusion is clear. Proving equality permits an elegant proof 
using the sandwich theorem [24, Theorem 4.1.19], which we sketch here. Take @ € 
d(f + g0T) (x) and assume without loss of generality that 


xb» f(x) + g(Tx) — o() 


attains a minimum of 0 at x. By Theorem 7 there is an affine function A := 
(T*y*, -) +r with —y* € dg(TX) such that 


T(x) —o(x) = Ax > —g(Ax). 


Equality is attained at x = X. It remains to check that dé + T* y* € df (x). a 
The next result is a useful extension to Proposition 2. 


Theorem 9 (Convexity and Regularity in Normed Spaces). Let f : X — 
(—co, +00] be proper and convex, and let x € dom f. The following are 
equivalent: 


(i) f is Lipschitz on some neighborhood of x. 
(ii) f is continuous at x. 
(iii) f is bounded on a neighborhood of x. 
(iv) f is bounded above on a neighborhood of x. 
(v) Of maps bounded subsets of X into bounded nonempty subsets of X*. 


Guide. See (24, Theorem 4.1.25]. ‘i 


The next results relate to Example 4 and provide additional tools for verifying dif- 
ferentiability of convex functions. The notation —,,« denotes weak* convergence. 


Theorem 10 (Smulian). Let the convex function f be continuous at X. 


(i) The following are equivalent: 
(a) f is Fréchet differentiable at X. 
(b) For each sequence x, — X and @ € df (xX), there existn € N and dy € 
Of (Xn) forn = 71 such that d, > ¢. 
(c) gn — @ whenever dy, € Of (Xn), & € Of (X). 
(ii) The following are equivalent: 
(a) f is Gateaux differentiable at X. 
(b) For each sequence x, — X and @ € df (xX), there existn € N and dy € 
Of (X,) forn = 71 such that dy >w* d. 
(c) bn >w* & whenever dy € Of (Xn), & € Of (X). 


A more complete statement of these facts and their provenance can be found in [24, 
Theorems 4.2.8 and 4.2.9]. In particular, in every infinite dimensional normed space, 
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there is a continuous convex function which is Gateaux but not Fréchet differentiable 
at the origin. 

An elementary but powerful observation about the subdifferential viewed as a 
multi-valued mapping will conclude this section. A multi-valued mapping T from 
X to X* is denoted with double arrows, T : X = X*. Then T is monotone if 


(v2 — v1, X2—x,) =O whenever v, € T(x,), v2 € T (x2). 
Proposition 6 (Monotonicity and Convexity). Let f : X — (—o0o,+c] be 
proper and convex on a normed space. Then the subdifferential mapping df : X = 
X* is monotone. 

Proof. Add the subdifferential inequalities in the Definition 2 applied to f(x,) and 
F (xo) for vy; € df (x1) and vo € OC f (x0). a 
3 Duality and Convex Analysis 

The Fenchel conjugate is to convex analysis what the Fourier transform is to 
harmonic analysis. To begin, some basic facts about this fundamental tool are 
collected. 


Fenchel Conjugation 


The Fenchel conjugate, introduced in [62], of a mapping f : X — [—oo, +co], as 
mentioned above is denoted f* : X* — [—oo, +00] and defined by 


Pix y= SIDE x) — f(x)}. 


The conjugate is always convex (as a supremum of affine functions). If the domain 
of f is nonempty, then f* never takes the value —oo. 


Example 5 (Important Fenchel Conjugates). 


(i) Absolute value. 


0 y €[-1,]] 


+oo else. 


f@)=l|x| @ER), f*O)= 


(ii) Lp norms (p > 1). 


* 1 
f@=sihl @>D, fO)=—bI (G+F=1), 


1 
P 
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In particular, note that the two-norm conjugate is “self-conjugate.” 
(ii1) Indicator functions. 


f=, f° =e, 


where oc is the support function of the set C. Note that if C is not closed and 
convex, then the conjugate of oc, that is the biconjugate of tc, is the closed 
convex hull of C.. (See Proposition 8(i1).) 

(iv) Boltzmann—Shannon entropy. 


_ yxinx-—x (x>0) 7 — 
foy=y (« =0)" f*y)=e” (yeR). 


(v) Fermi—Dirac entropy. 


xInx+(1—x)Ind-—x) (« €(0,1)) 
0 (x = 0,1) 


f*(y) = In. +e”) (y ER). 


fo) =| 


Some useful properties of conjugate functions are tabulated below. 


Proposition 7 (Fenchel-Young Inequality). Let X be anormed space and let f : 
X — [—0o, +00]. Suppose that x* € X* andx € dom f . Then 


FX) + FPR") = (x", x). (36) 
Equality holds if and only if x* € df (x). 


Proof sketch. The proof follows by an elementary application of the definitions 
of the Fenchel conjugate and the subdifferential. See [103] for the finite dimensional 
case. The same proof works in the normed space setting. a 

The conjugate, as the supremum of affine functions, is convex. In the following, 
the closure of a function f is denoted by f, and conv f is the function whose 
epigraph is the closed convex hull of the epigraph of /. 


Proposition 8. Let X be a normed space and let f : X — [—oo, +00]. 


(i) If f > g then g* > f*. 

di) ft =(F) = oars)’. 

Proof. The definition of the conjugate immediately implies (i). This immediately 
yields f* < (7) < (conv f)*. To show (ii) it remains to show that f* > 
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(conv f)*. Choose any ¢ € X*. If f*(¢) = +00 the conclusion is clear, so 
assume f*(~) = a for some a € R. Then ¢(x) — f(x) < a@ forall x € X. 
Define g := ¢ — f. Then g < conv ff and, by (i) (Conv f)* < g*. But g* =a, so 
(conv f)" <a = f*(). a 


Application of Fenchel conjugation twice, or biconjugation denoted by f**, is a 
function on X **. In certain instances, biconjugation is the identity — in this way, the 
Fenchel conjugate resembles the Fourier transform. Indeed, Fenchel conjugation 
plays a role in the convex analysis similar to the Fourier transform in harmonic 
analysis and has a contemporaneous provenance dating back to Legendre. 


Proposition 9 (Biconjugation). Let f : X — (—co, +00], x € X and x* € X*. 


OR mae Fae 2 
(ii) If f is convex and proper, then f**(x) = f(x) at x if and only if f is lsc at 
x. In particular, f is lsc if and only if fy* = f. 
(iii) f**|y = conv f ifconv f is proper. 


Guide. (i) follows from Fenchel—Young, Proposition 7, and the definition of the 
conjugate. (ii) follows from (i) and an epi-separation property [24, Proposition 
4.4.2]. (iii) follows from (ii) of this proposition and 8(ii). a 


The next results highlight the relationship between the Fenchel conjugate and the 
subdifferential already used in (28). 


Proposition 10. Let f : X — (—oo,+00] be a function and X € dom f. If 
gb € Of(X) then x € Of *(). If, additionally, f is convex and Isc at X, then the 
converse holds, namely X € Of *(f) implies @ € Of (X). 


Guide. See [72, Corollary 1.4.4] for the finite dimensional version of this fact that, 
with some modification, can be extended to normed spaces. oi 


Infimal convolutions conclude this subsection. Among them many applications 
are smoothing and approximation — just as is the case for integral convolutions. 


Definition 3 (Infimal Convolution). Let f and g be proper extended real-valued 
functions on a normed space X. The infimal convolution of f and g is defined by 


FO) = int FO) + a= 9): 


The infimal convolution of f and g is the largest extended real-valued function 
whose epigraph contains the sum of epigraphs of f and g; consequently, it is a 
convex function when f and g are convex. 
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The next lemma follows directly from the definitions and careful application of 
the properties of suprema and infima. 


Lemma 2. Let X be a normed space and let f and g be proper functions on X, 
then (fOg)* = f* + g*. 


An important example of infimal convolution is Yosida approximation. 


Theorem 11 (Yosida Approximation). Let f : X — R be convex and bounded 
on bounded sets. Then both fOn||- ||? and fOn|j-|| converge uniformly to f on 
bounded sets. 


Guide. This follows from the above lemma and basic approximation facts. a 


In the inverse problems literature ( f Gn|j- ||?) (0) is often referred to as Tikhonov 
regularization; elsewhere, f Cin|| - ||? is referred to as Moreau—Yosida regularization 
because fO 5 || - ||?, the Moreau envelope, was studied in depth by Moreau [94, 95]. 
The argmin mapping corresponding to the Moreau envelope — that is the mapping 
of x € X to the point y € X at which the value of fO 5 || - ||? is attained — is called 
the proximal mapping [94, 95, 107] 


. 1 
prox, p(x) = argmin yey f(y) + sy llx — y|P?. (37) 


When / is the indicator function of a closed convex set C, the proximal mapping is 
just the metric projection onto C, denoted by Pc (x): prox, ,.(x) = Pc(x). 


Fenchel Duality 


Fenchel duality can be proved by Theorem 5 and the sandwich theorem 7 [24, Theo- 
rem 4.4.18]. As the syllogism to this point deduces Fenchel duality as a consequence 
of the Hahn—Banach theorem. In order to close the Fenchel duality/Hahn—Banach 
circle of ideas, however, following [31] the main duality result of this section follows 
from the Fenchel—Young inequality and the next important lemma. 


Lemma 3 (Decoupling). Let X and Y be Banach spaces and let T : X — Y be 
a bounded linear mapping. Suppose that f : X — (—oo,+00] andg: Y > 


(—oo, +00] are proper convex functions which satisfy Assumption 6. Then there is 
ay* € Y* such that for any x € X andy €Y, 


p< (f()-(y*, Tx)) + (gQ) + 0%, y)), 


where p := infy{ f(x) + g(Tx)}. 
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Guide. Define the perturbed function h : Y — [—oo, +00] by 
h(u) := inft fOr) + g(x + w}, 


which has the property that h is convex, dom h = dom g —T dom /f and (the most 
technical part of the proof) 0 € int (dom h). This can be proved by assuming the 
first of the constraint qualifications (34). The second condition (35) implies (34). 
Then Theorem 5 yields 04(0) 4 ©, which guarantees the attainment of a minimum 
of the perturbed function. The decoupling is achieved through a particular choice of 
the perturbation u. See [31, Lemma 4.3.1]. HB 


One can now provide an elegant proof of Theorem 1, which is restated here for 
convenience. 


Theorem 12 (Fenchel Duality). Let X and Y be normed spaces, consider the 
functions f : X — (—c0o, +00] and g : Y > (—0oo0, +00] andletT : X > Y 
be a bounded linear map. Define the primal and dual values p, d € |—o0, +00] by 
the Fenchel problems 


p= inft fQ@) + g(Tx)} (38) 
d= up Pee Hs Er. (39) 
y*ey* 


These values satisfy the weak duality inequality p > d. 

If X,Y are Banach, f, g are convex and satisfy Assumption 6 then p = d, and 
the supremum to the dual problem is attained if finite. 
Proof. Weak duality follows directly from the Fenchel—Young inequality. 

For equality assume that p # —oo (this case is clear). Then Assumption 6 


guarantees that p < +00, and by the decoupling lemma (Lemma 3), there is a 
@ € Y* such that for all x € X¥ andy € Y 


PS (f(x) — (g, Tx)) + (g(y) — (-¢, y)- 
Taking the infimum over all x and then over all y yields 
pep OO -e -osasp. 
Hence, ¢ attains the supremum in (39), and p = d. a 


Fenchel duality for linear constraints, Corollary 1, follows immediately by 
taking g := Usp}. 
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Applications 


Calculus Fenchel duality is, in some sense, the dual space representation of the 
sandwich theorem. It is a straightforward exercise to derive Fenchel duality from 
Theorem 7. Conversely, the existence of a point of attainment in Theorem 12 yields 
an explicit construction of the linear mapping in Theorem 7: A := (T*@, -) +7, 
where ¢ is the point of attainment in (39) andr € [a, b] where a := infyex f(x) — 
(T*o, x) and b := sup,cy —g(Tz) — (T*¢, z). One could then derive all the 
theorems using the sandwich theorem, in particular the Hahn—Banach theorem 3 and 
the subdifferential sum rule, Theorem 8, as consequences of Fenchel duality instead. 
This establishes the Hahn—Banach/Fenchel duality circle: Each of these facts is 
equivalent and easily interderivable with the nonemptiness of the subgradient of a 
function at a point of continuity. 

An immediate consequence of Fenchel duality is a calculus of polar cones. Define 
the negative polar cone of a set K in a Banach space X by 


K~ = {x* € X* | (x*, x) <OVxeEK}. (40) 


An important example of a polar cone that has appeared in the applications discussed 
in this chapter is the normal cone of a convex set K at a point x € K, defined by 
(33). Note that 


Nx(X) = (K—-X). (41) 


Corollary 3 (Polar Cone Calculus). Let X and Y be Banach spaces and K Cc X 
and H CY be cones, and let A: X — Y bea bounded linear map. Then 


K~ + A*H” C(K+4'H) 


where equality holds if K and H are closed convex cones which satisfy H — AK = 
Y. 


This can be used to easily establish the normal cone calculus for closed convex sets 
C, and C2 at a point x € C); NC 


Noync (x) = No (x) + No, (x) 


with equality holding if, in addition, 0 € core (Cy — C2) or Cy N int Cp # @. 


Optimality Conditions Another important consequence of these ideas is the 
Pshenichnyi—Rockafellar [101, 103] condition for optimality for nonsmooth con- 
strained optimization. 
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Theorem 13 (Pshenichnyi—Rockafellar Conditions). Let X be a Banach space, 
let C Cc X be closed and convex, and let f : X — (—oo, +00] be a convex 
function. Suppose that either int C 0 dom f #4 @and f is bounded below on C, 
or C Mcont f # @ Then there is an affine functiona < f with infc f = infc a. 
Moreover, X is a solution to 


(Ps) minimize F(x) 
subject to xeEeCc 
if and only if 
0 € 0f(x%) + Nc(X). 
Guide. Apply the subdifferential sum rule to f + tc at X. a 


A slight generalization extends this to linear constraints 


minimize F(x) 
(Pin) _#€* 
subject to Tx e€D 


Theorem 14 (First-Order Necessary and Sufficient). Let X and Y be Banach 
spaces with D C Y convex, and let f : X — (—o0, +00] be convex andT : X > 
Y abounded linear mapping. Suppose further that one of the following holds: 
0 € core(D —T dom f), D is closed and f is Isc, (42) 
or 
T dom f Nint(D) 4 Z. (43) 
Then the feasible set C := {x € X |Tx € D} satisfies 
(fF + tc)(x) = Af(x) + T*(Npd(Tx)) (44) 
and X is a solution to (Pin) if and only if 
0 € af(X) + T*(Np(TX)). (45) 


A point y* € Y™® satisfying T*y* e€ —df(x) in Theorem 14 is a Lagrange 
multiplier. 


Lagrangian Duality The setting is limited to Euclidean space and the general 
convex program 


288 J.M. Borwein and D.R. Luke 


minimize fo(x) 
(Pevx) x€E 
subject to fix) <0 (fj =1,2,...,m) 
where the functions f; for 7 = 0,1,2,...,m are convex and satisfy 
() dom f; 4 ©. (46) 
= 


Define the Lagrangian L: E x R"4 — (—oo, +00] by 
L(x, A) := fo(x) +47 F(x), 


where F := (fi, fo,..., fm)’. A Lagrange multiplier in this context is a vector 
AX € Rx for a feasible solution ¥ if ¥ minimizes the function L(-,A) over 
E and XQ satisfies the so-called complimentary slackness conditions: Aj = 0 
whenever fj (x) < 0. On the other hand, if X is feasible for the convex program 
(Pcyx) and there is a Lagrange multiplier, then x is optimal. Existence of the 
Lagrange multiplier is guaranteed by the following Slater constraint qualification 
first introduced in the 1950s. 


Assumption 7 (Slater Constraint Qualification). There exists an xX € dom fowith 
Fi (&) < Ofor j =1,2,...,m. 


Theorem 15 (Lagrangian Necessary Conditions). Suppose that x € dom fo is 
optimal for the convex program (P-yx) and that Assumption 7 holds. Then there is a 
Lagrange multiplier vector for x. 


Guide. See [27, Theorem 3.2.8]. fet 


Denote the optimal value of (Pcyx) by p. Note that, since 


f(x) ifx €edom f 


sup L(x,A) = 
ERM +oo otherwise, 
then 
p= inf sup L(x,A). (47) 
xE€E AER” 4. 


It is natural, then to consider the problem 


d= sup inf L(x,A) (48) 
AER 4. x€E 
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where d is the dual value. It follows immediately that p > d. The difference 
between d and p is called the duality gap. The interesting problem is to determine 
when the gap is zero, that is, when d = p. 


Theorem 16 (Dual Attainment). /f Assumption 7 holds for the convex program- 
ming problem (P-yx), then the primal and dual values are equal and the dual value 
is attained if finite. 


Guide. For a more detailed treatment of the theory of Lagrangian duality see [27, 
Sect. 4.3]. B 


Optimality and Lagrange Multipliers 


In the previous sections, duality theory was presented as a byproduct of the Hahn— 
Banach/Fenchel duality circle of ideas. This provides many entry points to the 
theory of convex and variational analysis. For present purposes, however, the 
real significance of duality lies with its power to illuminate duality in convex 
optimization, not only as a theoretical phenomenon but also as an algorithmic 
Strategy. 

In order to get to optimality criteria and the existence of solutions to convex 
optimization problems, turn to the approximation of minima, or more generally the 
regularity and well-posedness of convex optimization problems. Due to its reliance 
on the Slater constraint qualification (49), Theorem 16 is not adequate for problems 
with equality constraints: 


minimize 0(x) 
(Peg) *68 - 
subject to F(x) =0 


for S C E closed and F: E + Y a Fréchet differentiable mapping between the 
Euclidean spaces F and Y. 
More generally, one has problems of the form 


minimize fo(x) 
xeS 


(Pe) (49) 


subject to F(x)eD 


for E and Y Euclidean spaces, and S C E and D C Y are convex but not 
necessarily with nonempty interior. 


Example 6 (Simple Karush-Kuhn-Tucker). For linear optimization problems, rel- 
atively elementary linear algebra is all that is needed to assure the existence of 
Lagrange multipliers. Consider 
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minimize fo(x) 
xe 
subject to fi(x)€D;, j =1,2,...,m 


for fj: R” > R Gj =0,1,2,..., 5) continuously differentiable, f; : R" > R 
(j = s+ 1,...,m) linear. Suppose S C E is closed and convex, while D; := 
(—o0, 0] for 7 = 1,2,...,s and D; := {0} for 7 =s+1,...,m. 


Theorem 17. Denote by fj(x) the submatrix of the Jacobian of (fi,..., iy 
(assuming this is defined at x) consisting only of those fi for which fj(x) = 0. 
In other words, f(x) is the Jacobian of the active inequality constraints at x. 
Let x be a local minimizer for (Pz) at which f; are continuously differentiable 


(j =0,1,...,5) and the matrix 
er) ” 


is full-rank where A := (V fy4t...-+V fm)!» Then there are X € R° and Ti € R” 
satisfying 


A= 0. (Sla) 
(AG) icy fA =O (51b) 
fo) + Af) + "A = 0. (Slc) 
j=l 
Guide. An elegant and elementary proof is given in [36]. a 


For more general constraint structure, regularity of the feasible region is essential 
for the normal cone calculus which plays a key role in the requisite optimality 
criteria. More specifically, one has the following constraint qualification. 
Assumption 8 (Basic Constraint Qualification). 

y = (0,...,0) is the only solution in Np(F(x)) to0 € VF! (x)y + Ns(x). 
Theorem 18 (Optimality on Sets with Constraint Structure). Let 

C={xeS|F(x)€ D} 
for F = (fi, fa,..., fm) + E = R" with f; continuously differentiable (j = 
1,2,...,m), S C E closed, and for D = D, x D2 x-+» Dm C R" with Dj; closed 


intervals (j = 1,2,..., m). Then for any xX € C at which Assumption 8 is satisfied 
one has 
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Nc (X) = VF" (®)Np(F()) + Ns (x). (52) 


If, in addition, fo is continuously differentiable and X is a locally optimal solution 
to (Pg) then there is a vector y € Np(F(x)), called a Lagrange multiplier such 
that 0 € V fo(x) + VF! (x)¥ + Ns (X). 


Guide. See [107, Theorems 6.14 and 6.15]. | 


Variational Principles 


The Slater condition (49) is an interiority condition on the solutions to optimization 
problems. Interiority is just one type of regularity required of the solutions, wherein 
one is concerned with the behavior of solutions under perturbations. The next 
classical result lays the foundation for many modern notions of regularity of 
solutions. 


Theorem 19 (Ekeland’s Variational Principle). Let (X,d) be a complete metric 
space and let f : X — (—oo, +00] be a Isc function bounded from below. Suppose 
thate > Oandz€ X satisfy 


I@® < inf f +e. 


For a given fixed i > 0, there exists y € X such that 


(i) dy) <A. 
(ii) f(y) + fd(zy) < f(. 
(iii) f(x) + d(x, y) > f(y), for all x € X\{y}. 


Guide. For a proof see [61]. For a version of the principle useful in the presence of 
symmetry see [32]. @ 


An important application of Ekeland’s variational principle is to the theory of 
subdifferentials. Given a function f : X — (—co, +00], a point x9 € dom f and 
€ > 0, the e-subdifferential of f at xo is defined by 


Je f (x0) = {6 € X* | (b,x — x0) < f(x) — f(xo) +e, Vx € X}. 


If x9 ¢ dom f then by convention 0, f(xo) := @. When € = 0 one has 0, f(x) = 
df (x). For « > 0 the domain of the ¢-subdifferential coincides with dom f when 
f is a proper convex Isc function. 


Theorem 20 (Brgnsted—Rockafellar). Suppose f is a proper lsc convex function 
ona Banach space X. Then given any xy € dom f,¢ > 0,A > O and wo € 0¢ f (xo) 
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there exist x € dom f and w € X* such that 


wedf(x), ||x—xo|| <€/A and ||w—woll <A. 


In particular, the domain of Of is dense in dom f. 


Guide. Define g(x) := f(x) — (wo, x) on X, a proper Isc convex function with 
the same domain as f. Then g(xo) < infy g(x) + €. Apply Theorem 19 to yield a 
nearby point y that is the minimum of a slightly perturbed function, g(x)+A||x—y]. 
Define the new function h(x) := A||x — y|| — g(y), so that h(x) < g(x) forall X. 
The sandwich theorem (Theorem 7) establishes the existence of an affine separator 
a + ¢ which is used to construct the desired element of df(x). a 


A nice application of Ekeland’s variational principle provides an elegant proof of 
Klee’s problem in Euclidean spaces [75]: Is every Cebyéev set C convex? Here, a 
Cebyéev set is one with the property that every point in the space has a unique best 
approximation in C. A famous result is as follows. 


Theorem 21. Every Cebyéev set in a Euclidean space is closed and convex. 


Guide. Since, for every finite dimensional Banach space with smooth norm, 
approximately convex sets are convex, it suffices to show that C is approximately 
convex, that is, for every closed ball disjoint from C there is another closed ball 
disjoint from C of arbitrarily large radius containing the first. This follows from the 
mean value Theorem 6 and Theorem 19. See [24, Theorem 3.5.2]. It is not known 
whether the same holds for Hilbert space. i 


Fixed Point Theory and Monotone Operators 
Another application of Theorem 19 is Banach’s fixed point theorem. 


Theorem 22. Let (X,d) be a complete metric space and let @ : X — X . Suppose 
there isa y € (0,1) such that d(¢(x),¢(y)) < yd(x, y) for all x,y € X. Then 
there is a unique fixed point x € X such that $(X) =X. 


Guide. Define f(x) := d(x, ¢(x)). Apply Theorem 19 to f with A = 1 and 
€ = 1—y. The fixed point X satisfies f(x) + ed(x,x) => f(x) forallxe X. Hf 


The next theorem is a celebrated result in convex analysis concerning the 
maximality of lsc proper convex functions. A monotone operator T on X is maximal 
if gph T cannot be enlarged in X x X without destroying the monotonicity of T. 
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Theorem 23 (Maximal Monotonicity of Subdifferentials). Let f : X —- 
(—oo, +00] be alsc proper convex function on a Banach space. Then of is maximal 
monotone. 


Guide. The result was first shown by Moreau for Hilbert spaces [95, Propo- 
sition 12.b], and shortly thereafter extended to Banach spaces by Rockafellar 
[102, 104]. For a modern infinite dimensional proof see [1, 24]. This result fails 
badly in incomplete normed spaces [24]. a 


Maximal monotonicity of subdifferentials of convex functions lies at the heart 
of the success of algorithms as this is equivalent to firm nonexpansiveness of 


the resolvent of the subdifferential (J + df)~! [92]. An operator T is firmly 
nonexpansive on a closed convex subset C C X when 


Tx —Ty|? <(x-—y,Tx—Ty) forallx,y € X. (53) 
T is just nonexpansive on the closed convex subset C C X if 
Tx —Ty|| < |lx-—yl]| forallx,y EC. (54) 
Clearly, all firmly nonexpansive operators are nonexpansive. One of the most 
longstanding questions in geometric fixed point theory is whether a nonexpansive 
self-map T of a closed bounded convex subset C of a reflexive space X must have 
a fixed point. This is known to hold in Hilbert space. 


4 Case Studies 


One can now collect the dividends from the analysis outlined above for problems of 
the form 


minimize Ig(x) 
xECCX (55) 
subject to Ax €D 


where X and Y are real Banach spaces with continuous duals X* and Y*, C and D 
are closed and convex, A: X — Y is acontinuous linear operator, and the integral 
functional I,(x) := J; p(x(t))(dt) is defined on some vector subspace L ,(T, j1) 
of X. 


Linear Inverse Problems with Convex Constraints 


Suppose X is a Hilbert space, D = {b} € R” and g(x) := 5||x|?. To apply Fenchel 
duality, rewrite (12) using the indicator function 
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minimize \x|[? + tc(x) 
nimize —SIlx|? + wc a 
subject to Ax =b. 


Note that the problem is posed on an infinite dimensional space, while the 
constraints (the measurements) are finite dimensional. Here Fenchel duality is used 
to transform an infinite dimensional problem into a finite dimensional problem. Let 
F := {x €C CE | Ax = )b} and let G denote the extensible set in FE consisting 
of all measurement vectors b for which F is nonempty. Potter and Arun show that 
the existence of y € R” such that b = APcA*y is guaranteed by the constraint 
qualification b € ri G, where ri denotes the relative interior [100, Corollary 2]. This 
is a special case of Assumption 6, which here reduces to b € int A(C). Though at 
first glance the latter condition is more restrictive, it is no real loss of generality 
since, if it fails, one can restrict the problem to range(A) which is closed. Then it 
turns out that b € Aqri C, the image of the quasi-relative interior of C [27, Exercise 
4.1.20]. Assuming this holds Fenchel duality, Theorem 12, yields the dual problem 


sup (b, y) — (4) Il? tec)” (A*y), (57) 


yeRn 
whose value is equivalent to the value of the primal problem. This is a finite dimen- 


sional unconstrained convex optimization problem whose solution is characterized 
by the inclusion (Theorem 4) 


0€ (AI? tec) (A*y) —5. (58) 


Now from Lemma 2, Examples 5(ii) and (iii), and (37), 


(SIP + tc)” Ge) = (ac OZ Il) () = inf oc(@ + alle — al? 


The argmin of the Yosida approximation above (see Theorem 11) is the proximal 
operator (37). Applying the sum rule for differentials, Theorem 8 and Proposition 10 
yield 


Prox, 5 (x) = argmin cy {oc (2) + 4|lz—xIl?} = x — Pc(x), (59) 


where Pc is the orthogonal projection onto the set C. This together with (58) yields 
the optimal solution y to (57): 


b = APc(A*7). (60) 


Note that the existence of a solution to (60) is guaranteed by Assumption 6. This 
yields the solution to the primal problem as X = Pc (A*Y). 

With the help of (59), the iteration proposed in [100] can be seen as a subgradient 
descent algorithm for solving 
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inf, h(y) == oc(A"y — Pc(A*y)) + zllPe(A* y)II” — (b, y) 


The proposed algorithm is, given yo € R” generates the sequence {y, }°2.) by 
Yn+1 = Yn — Adh(yn) = Vn t+ A (b — APc A* yn) : 


For convergence results of this algorithm in a much larger context see [55]. 


Imaging with Missing Data 


This application is formally simpler than the previous example since there is no 
abstract constraint set. As discussed in section “Imaging with Missing Data” in 
Sect. | relaxations to the conventional problem take the form 


minimize Tox (x) 
xeER" eL (61) 
subject to Ax = b, 
where 
* € x. € 
1%) = mn In (4*4/¢ 4 1) —xL—e, (62) 


Using Fenchel duality, the dual to this problem is the concave optimization problem 


sup yb — Ip, ,(A*y), 


ye Rn 


where 


_ (L+x)In(L+x)+(L—x)In(L—x)  In(Z) 
Pet (X) = e( 2L In(2) 7 ny) 
L,e>Ox € [-L, L]. 


If there exists a point y satisfying b = AA*Y, then the optimal value in the dual 
problem is attained and the primal solution is given by A*y. The objective in the 
dual problem is smooth and convex, so we could apply any number of efficient 
unconstrained optimization algorithms. Also, for this relaxation, the same numerical 
techniques can be used for all L — 0. For further details see [28]. 
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Inverse Scattering 


Theorem 24. Let X,Y be reflexive Banach spaces with duals X* and Y*. Let F : 
Y* > Y andG: X — Y be bounded linear operators with F = GSG* for 
S.: X* — X abounded linear operator satisfying the coercivity condition 


I(v, Sv)| = cllelly* for some c > 0 and all g € range(G*) C X*. 


Define h(w) : Y* — (—oo, +00] := |(w, Fw)|, and let h* denote the Fenchel 
conjugate of h. Then range(G) = dom h*. 


Proof. Following [74, Theorem 1.16], one shows that h*(@) = oo ford ¢ 
range(G). To do this it is helpful to work with a dense subset of range G: G*(C) 
forC := {w € Y* | (wv, @) = 0}. It was shown in [74, Theorem 1.16] that G*(C) 
is dense in range(G). 

Now by the Hahn—Banach theorem 3 there is a b € Y* such that (6. 0) =1. 
Since G*(C) is dense in range(G*), there is a sequence {y%,}92, C C with 


n=1 
Grn > -G*¢, n—> oo. 


Now set Wy, := Vn +¢. Then (¢, aw,) = a and G*(ay,) = aG*y, — 0 for any 
a € R. Using the factorization of F one has 


(Yn, Yn) | = |(G* Yn, SG* Vn) | < SING" vn lies 


hence a? (Wy, FWn) > 0 asn —> ov for all a, but (¢, a) = a, that is, 
(f,aWn) —h(aWn) > a and h*(p) = ov. a 


In the scattering application, one has a scatterer supported on a domain D C 
R” (m = 2 or 3) that is illuminated by an incident field. The Helmholtz equation 
models the behavior of the fields on the exterior of the domain and the boundary data 
belongs to X = H'/?(I). On the sphere at infinity the leading-order behavior of 
the fields, the so-called far field pattern, lies in Y = L?(S). The operator mapping 
the boundary condition to the far field pattern — the data-to-pattern operator — is 
G: H'/?(C) > L?(S). Assume that the far field operator F : L?(S) > L?(S) 
has the factorization F = GS*G*, where S : H~'/?(T) > H'/?(L) isa single 
layer boundary operator defined by 


(Sp) (x) = [ (x, y)y(y)ds(y), x ET, 


for ®(x, y) the fundamental solution to the Helmholtz equation. With a few results 
about the denseness of G and the coercivity of S, which, though standard, will be 
glossed over, one has the following application to inverse scattering. 
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Corollary 4 (Application to Inverse Scattering). Let D C R” (m = 2 or 3) 
be an open bounded domain with connected exterior and boundary 1. Let G : 
H'/(C) + L?(S), be the data-to-pattern operator, S : H~'/?(T) > H'/(T), 
the single layer boundary operator and let the far field pattern F : L?(S) > L(S) 
have the factorization F = GS*G*. Assume k? is not a Dirichlet eigenvalue of —A 
on D. Thenrange G = dom h* where h() : L?(S) > (—00, +00] := |(W, FW)|. 


Fredholm Integral Equations 


The introduction featured the failure of Fenchel duality for Fredholm integral 
equations. The following sketch of a result on regularizations, or relaxations, 
recovers duality relationships. The result will show that by introducing a relaxation, 
one can recover the solution to ill-posed integral equations as the norm limit of 
solutions computable from a dual problem of maximum entropy type. 


Theorem 25 ([22], Theorem 3.1). Let X = L,(T,) on a complete mea- 
sure finite measure space and let (Y,||-||) be a normed space. The infimum 
inf,cy {Ip(x) | Ax = b} is attained when finite. In the case where it is finite, 
consider the relaxed problem for € > 0 


minimize I(x) 
(Pur) ex ‘ 
subject to || Ax — || <. 


Let p, denote the value of (cee The value of p. equals d_-, the value of the dual 
problem 


(Poep) maximize (b, y*) —€lly" ll — Ips(A*y"), 
y 


and the unique optimal solution of (Piru) is given by 


ag* 
or 


= KK 

Xo. c= (A Ve ) , 

where y; is any solution to (Pina): Moreover, as € — 0*, Xg.< converges in mean 
to the unique solution of (Pace) and pe > Po. 


Guide. Attainment of the infimum in inf,<y { Ig(x) | Ax = b} follows from strong 
convexity of Ig [26, 112]: strictly convex with weakly compact lower-level sets and 
with the Kadec property, i.e., that weak convergence together with convergence 
of the function values implies norm convergence. Let g(y) := ts(y) for S = 
{y €Y |b ey + By} and rewrite (P{,,p) as inf {1,(x) + g(Ax) |x € Xx}. An 
elementary calculation shows that the Fenchel dual to (Puen) is (Ps var The 
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relaxed problem (Pir E p) has a constraint for which a Slater-type constraint 
qualification holds at any feasible point for the unrelaxed problem. The value d, is 
thus attained and equal to p,. Subgradient arguments following [25] show that Xo. 
is feasible for (Pir E p) and is the unique solution to (Per E mh Convergence follows 
from weak compactness of the lower-level set L(po) := {x | Ig(x) < po 13 which 
contains the sequence (Xg.<)es9. Weak convergence of X,,- to the unique solution 
to the unrelaxed problem follows from strict convexity of J,. Convergence of the 
function values and strong convexity of J, then yields norm convergence. si 


Notice that the dual in Theorem 25 is unconstrained and easier to compute with, 
especially when there are finitely many constraints. This theorem remains valid for 
objectives of the form I(x) + (x*, x) for x* in Loo(T). This enables one to apply 
them to many Bregman distances, that is, integrands of the form @(x) — $(xo) — 
($' (x0), x — Xo), where ¢ is closed and convex on R. 


5 Open Questions 


Regrettably, due to space constraints, fixed point theory and many facts about 
monotone operators that are useful in proving convergence of algorithms have been 
omitted. However, it is worthwhile noting two long-standing problems that impinge 
on fixed point and monotone operator theory. 


1. Klee’s problem: is every Cebyéev set C in a Hilbert space convex? 
2. Must a nonexpansive self-map 7 of a closed bounded convex subset C of a 
reflexive space X have a fixed point? 


6 Conclusion 


Duality and convex programming provides powerful techniques for solving a wide 
range of imaging problems. While frequently a means toward computational ends, 
the dual perspective can also yield new insight into image processing problems and 
the information content of data implicit in certain models. Five main applications 
illustrate the convex analytical approach to problem solving and the use of duality: 
linear inverse problems with convex constraints, compressive imaging, image 
denoising and deconvolution, nonlinear inverse scattering, and finally Fredholm 
integral equations. These are certainly not exhaustive, but serve as good templates. 
The Hahn—Banach/Fenchel duality cycle of ideas developed here not only provides 
a variety of entry points into convex and variational analysis, but also underscores 
duality in convex optimization as both a theoretical phenomenon and an algorithmic 
Strategy. 

As readers of this volume will recognize, not all problems of interest are convex. 
But just as nonlinear problems are approached numerically by sequences of linear 
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approximations, nonconvex problems can be approached by sequences of convex 
approximations. Convexity is the central organizing principle and has tremendous 
algorithmic implications, including not only computable guarantees about solutions, 
but efficient means toward that end. In particular, convexity implies the existence 
of implementable, polynomial-time, algorithms. This chapter is meant to be a 
foundation for more sophisticated methodologies applied to more complicated 
problems. 
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1 Maximum Likelihood Estimation 


Expectation-maximization algorithms, or EM algorithms for short, are iterative 
algorithms designed to solve maximum likelihood estimation problems. The general 
setting is that one observes a random sample Yj, Y2,..., Y, of a random variable Y 
whose probability density function (pdf) f(- |x.) with respect to some (known) 
dominating measure is known up to an unknown “parameter” x,. The goal is to 
estimate x, and, one might add, to do it well. In this chapter, that means to solve the 
maximum likelihood problem 


maximize [| f(¥;|x) over x, (1) 
i=l 


and to solve it by means of EM algorithms. The solution, assuming it exists and is 
unique, is called the maximum likelihood estimator of x,. Here, the estimator is 
typically denoted by x. 

The notion of EM algorithms was coined by [27], who unified various earlier 
instances of EM algorithms and in particular emphasized the notion of “missing” 
data in maximum likelihood estimation problems, following Hartley [53]. Here, the 
missing data refers to data that were not observed. Although this seems to imply that 
these data could have or should have been observed, it is usually the case that these 
missing data are inherently inaccessible. Typical examples of this are deconvolution 
problems, but it may be instructive to describe a simplified version in the form of 
finite mixtures of probability densities. 

Let the random variable Y be a mixture of some other continuous random 
variables Z,, Z2,..., Zm for some known integer m. For each j, 1 < j < m, 
denote the pdf of Z; by f(- | 7). The pdf of Y is then 
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fro =) wi fo 1D. (2) 
j=l 


where w* = (w*, W3,-- w*) is a probability vector. In other words, the ws are 


a) m 


nonnegative and add up to 1. The interpretation is that for each /, 
Y = Z;_ with probability w;. (3) 


As before, given a random sample Y, Y,..., Y, of the random variable Y, the 
goal is to estimate the “parameter” w*. The maximum likelihood problem for doing 
this is 


maximize il | y wy f% |S) (4) 


i=1 


j=l 


subject to w= (w1,W2,...,Wm) isa probability vector. 


Now, what are the “missing” data for this finite mixture problem? In view of 
the interpretation (3), it would clearly be useful if for each Y;, it was known which 
random variable Z; it was supposed to be a random sample of. Thus, let J; € 
{1,2,...,m} such that 


ili SZ. (5) 
Then, J), J2,..., J, would be a random sample of the random variable J, whose 
distribution would then be easy to estimate: for each /, 
#{J; = j} 
re ae (6) 


the fraction of the J; that were equal to 7. Note that the distribution of J is given by 
P[J = j] =wj, JH 1, 2y..55M: (7) 


Of course, unfortunately, the J; are not known. It is not even apparent that it would 
be advantageous to think about the J;, but in fact it is, as this chapter tries to make 
clear. 

From the image processing point of view, the above problem becomes more 
interesting if the finite sum in (2) is interpreted as a discretization of the integral 
transform 


fy) = f FOL we) ax (8) 


and the goal is to recover the function or image w from the random sample 
Y,, Yo,..., Y,. The maximum likelihood problem of estimating w, 
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maximize l | | fa% | x) w(x) dx over w, (9) 


i=1 


is (formally) a straightforward extension of the mixture problem. Such (one- and 
two-dimensional) deconvolution problems abound in practice, e.g., in astronomy 
and tomography. See the suggested reading list. 

In the next two sections, the bare essentials of a more-or-less general version 
of the EM algorithm are presented and it is shown that it increases the likelihood. 
Without special conditions, that is, all one can say about the convergence of the EM 
algorithm, one cannot even claim that in the limit, it achieves the maximum value 
of the likelihood. See [98]. For the convex case, where the negative log-likelihood 
is convex and the constraint set is convex as well, one can say much more, as will 
become clear. 

Before discussing the two “big” applications of positron emission tomography 
(PET) and three-dimensional electron microscopy (3D-EM. Yes, another instance 
of EM!), it is prudent to discuss some simple examples of maximum likelihood 
estimation and to derive the associated EM algorithms. It turns that the prototypical 
example is that of estimating the weights in a mixture of known distributions; 
see (2). By analogy, this example shows how one should derive the EM algorithm 
for deconvolution problems with binned data, which is similar to the situation 
in positron emission tomography. The general parametric maximum likelihood 
estimation is also discussed, as well as the related case of empirical Bayes 
estimation. The latter has some similarity with 3D-EM. 

All of this naturally leads to the discussion of the maximum likelihood approach 
to positron emission tomography (which originated with Rockmore and Macovski 
[82], but who mistakenly took the road of a least-squares treatment) and the EM 
algorithm of Shepp and Vardi [88]. This is one of the classic examples of Poisson 
data. However, even Poisson data may be interpreted as a random sample of some 
random variable; see section “The Emission Tomography Experiment.” For the 
ubiquitous nature of Poisson data, see [4] and references therein. 

The very messy example of the reconstruction of the shapes of macromolecules 
of biological interest by way of 3D-EM also passes review. 

For the example of mixtures of known distributions as well as for positron 
emission tomography, there is a well-rounded theory for the convergence of the 
EM algorithm to wit the alternating projection approach of Csiszar and Tusnddy 
[23] and the majorizing functional approach of Miilthei and Schorr [75] and De 
Pierro [29]. This approach extends to EM-like algorithms for some maximum 
likelihood-like problems. Unfortunately, this ignores the fact that the maximum 
likelihood problem is ill conditioned when the number of components in the 
mixture is large (or that the deconvolution problem is ill-posed). So, one needs 
to regularize the maximum likelihood problems, and then, in this chapter, the 
issue is whether there are EM algorithms for the regularized problems. For the 
PET problem, this certainly works for Bayesian approaches, leading to maximum 
a posteriori (MAP) likelihood problems as well as to arbitrary convex maximum 
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penalized likelihood problems. In this context, mention should be made of the EMS 
algorithm of Silverman et al. [90], the EM algorithm with a linear smoothing step 
added, and the NEMS algorithm of Eggermont and LaRiccia [36] in which an extra 
nonlinear smoothing step is added to the EMS algorithm to make it a genuine EM 
algorithm for a smoothed maximum likelihood problem. However, the convergence 
of regularization procedures for ill-posed maximum likelihood estimation problems, 
whether Tikhonov style penalization or “optimally” stopping the EM algorithm, will 
not be discussed. See, e.g., [80]. 

The final issue under consideration is that EM algorithms are painfully slow, 
so methods for accelerating EM algorithms are discussed as well. The accelerated 
methods take the form of block-iterative methods, including the extreme case of 
row-action methods. 

The selection of topics is driven by applications to image processing. As such, 
there is very little overlap with the extensive up-to-date survey of EM algorithms of 
McLachlan and Krishnan [71]. 


2 The Kullback-Leibler Divergence 


Before turning to the issue of EM algorithms, emphatic mention must be made 
of the pervasiveness of the Kullback—Leibler divergence (also called /-divergence 
or information divergence; see, e.g., [22] and references therein) in maximum 
likelihood estimation. For probability density functions f and g on R% say, it is 
defined as 


f(y) 


K(f) =f (40 tog | a 


poi fo) PMO (10) 


with denoting Lebesgue measure. Here, 0 log(0/0) is defined as 0. Note that the 
Kullback—Leibler divergence is not symmetric in its arguments. Also note that the 
integrand is nonnegative, so that the integral is well defined if the value +00 is 
admitted. Moreover, the integrand equals 0 if and only if f(y) = g(y), so that 
KL(f, g) > Ounless f = g almost everywhere, in which case KL(f, g) = 0. 

Now consider the problem of estimating the unknown parameter x, in a 
probability density f(- |x). In view of the above, the ideal way would be to 


minimize KL(f(- |xo), f(-|x)) over x, (11) 


but of course, this is not a rational problem because the objective function is 
unknown. However, note that 


KL(F(« bro) fC [s)) =--{. Ole.) log FO|x) duty) 


+ / Fx) log fOrlx.) du(y), 2) 
Rd 
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and that the second term does not depend on x. So, the problem (11) is equivalent 
to (has the same solutions as) 


minimize -{ F(y|Xo) log f(y|x) duty) over x. (13) 
Rd 


Of course, this is still not a rational problem, but the objective function equals 
4 [Ln (x)], where L,,(x) is the scaled negative log-likelihood 


1 n 
Ly(x) =—= ) | log f(%i|x), (14) 


i=l 


if Y|, Y,...,Y, is a random sample of the random variable Y with probability 
density function f( - |x,). So, solving the maximum likelihood problem (1) may 
be viewed as approximately solving the minimum Kullback—Leibler divergence 
problem (11). This is the basic reason for the pervasiveness of the Kullback—Leibler 
divergence in the analysis of maximum likelihood estimation and EM algorithms. 

The above illustrates two additional points. First, in maximum likelihood esti- 
mation, one attempts to solve the minimum Kullback—Leibler problem (11) by first 
estimating the objective function. So, if the estimator is “optimal” at all, it has to be 
in a sense related to the Kullback—Leibler divergence. Second, one may well argue 
that one is not estimating the parameter x, but rather the density f(- |x,). This 
becomes especially clear if f( - |x) is reparametrized as g( - |z) = f ( - |T(z)) for 
some transformation T. This would have an effect on the possible unbiasedness of 
the estimators X and Z of x, and z,. However, under reasonable conditions on 7’, the 
maximum likelihood density estimators f(- |) and y(- |Z) of f(- | xo) will be 
the same. 


3 The EM Algorithm 
The Maximum Likelihood Problem 


Let (Y, By, P) be a statistical space, i-e., (7, By) is a measurable space and P is a 
collection of probability measures on By, represented as a family indexed by some 
index set ¥ as follows: 


P= {P(- |x): x € X}. (15) 


Assume that there is a measure P.. that dominates P in the sense that every P( - |x) 
is absolutely continuous with respect to Poo. Then, the Radon—Nikodym derivative 
of P(- |x) with respect to Poo exists and is By measurable for all x € ¥. It may 
be written as 


dP(- |x) 


Ay Ox) = [| (y) (16) 
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and is referred to as the density of P( - |x) with respect to Poo. It should be observed 
that fy ( - |X.) = fy(- ) is the density of the random variable Y with respect to 
Poo. For arbitrary x, fy (- |x) is a density, but since it is not known of what random 
variable, the subscript V is used here. 

Let Y be a random variable with values in )V, and assume that it is distributed 
as P(-|x,) for some (unknown) x, € %. The objective is to estimate x, based 
on a random sample Y), Y2,..., Y, of the random variable Y. Note that estimating 
X» amounts to constructing a measurable function of the data, which may then be 
denoted as ¥ = X(¥;, Yo,...,Y,). 

The maximum likelihood problem for estimating x, is then written in the form 
of minimizing the scaled negative log-likelihood, 


n 1 n 
minimize L,(x) = = Y> log fy (Yilx) (17) 


i=1 


subjectto x EX. 


In this formulation, the parameter x is deemed important. The alternative formula- 
tion in which the densities are deemed important is 


cing def 1 ” 
minimize L,(f) = > > log f(%) (18) 
i=l 


subjectto f EP. 


In this formulation, there are two ingredients: the likelihood function and the 
(parametric) family of densities under consideration. 

It is not obvious that solutions should exist, especially if the index set V is 
large, but in applications of image processing type, this turns out to be of lesser 
importance than one might think. See Sect. 7. Regardless, closed form solutions are 
generally not available, and one must employ iterative methods for the solution of 
the maximum likelihood problem. In this chapter, that means EM algorithms. 


The Bare-Bones EM Algorithm 


Here, the bare essentials of the EM algorithm are presented. The basic premise in 
the derivation of the EM algorithm is that there is “missing” data that would make 
estimating x, a lot easier had they been observed. So, assume that the missing data 
refers to data in a space Z, with (Z, Bz, Q) another statistical space, where the 
collection of probability measures is again indexed by 7X, 


O={Q(- |x): xe X}. (19) 
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Assume that Q is dominated by some measure Qo, and denote the associated 
Radon—Nikodym derivatives as 


dQ(- |x) 


Fz (2|x) = feo 


(z), zEZ. (20) 


Let Z be a random variable with values in Z and with distribution Q(- |x,), with 
the same x, as for the random variable Y. Note that the pair (Y, Z) takes on values 
in Y x Z. The relevant statistical space is then (V x Z, Byxz,R), where Byxz 
is the smallest o-algebra that contains all sets A x B with A € By and B € Bz. 
Again, assume that R may be indexed by -V as 


R= {R(- |x): x € ¥}, (21) 
and that is dominated by some measure Ro. Write 


dR(- |x) 


Fy zy. 21x) = || (y, 2) (22) 


for the associated Radon—Nikodym derivatives. 

Now, if the “complete” data (V1, Z,), (V2, Z2),..., (Yn, Zn), a random sample 
of the random variable (Y, Z), is available, then the maximum likelihood problem 
for estimating X, is 


1 n 
minimize — — log fy, 2%, Zi|x) 
n 2 soe (23) 


subjectto x Er. 


Of course, this is not a rational problem, since the Z; went unobserved. In other 
words, the objective function is not known (and not knowable). However, one may 
attempt to estimate it by the conditional expectation 

“| 


where Y, = (Y, Y2,...,¥Y,). The fly in the ointment is that computing this 
expectation involves the distribution of Z conditioned on Y, which surely will 
involve the unknown x, one wishes to estimate! So, at this point, assume that 
some initial guess x; for x, is available; then denote the resulting (approximate) 
conditional expectation by E[...|Yn, x1]. 

Determining this conditional expectation constitutes the E-step of the first 
iteration of the EM algorithm. The M-step of the first iteration then amounts to 
solving the minimization problem 


1 n 
| ee | ef 20%. 20) 


i=1 


| iix 
Ms "a > log fy z(V%i, Zi|x) 


i=1 
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1 
minimize Ay(x|x1) = —— 2 Blog fy, 5%. Zilx) |x, Yi] oa 


subjectto xe XX. 


Denote the solution by x2 (assuming it exists). Suppressing the presence of the Y; 
in the notation, one may define the iteration operator by x2 = R(x,), and then the 
EM algorithm may be stated as 


Xk+1 = R(xx), k = 12. ey (25) 


provided x; has been chosen appropriately. This is the bare-bones version of the EM 
algorithm. Note that it may not be necessary to solve the problem (24) exactly, e.g., 
one may consider (over)relaxation ideas or the so-called stochastic EM algorithms. 
See, e.g., [76] and references therein. This will not be considered further. 


Remark I. (a) It may be inappropriate to speak of the EM algorithm, since the 
introduction of different missing data may lead to a different algorithm. 
However, usually there is not much choice in the missing data. 

(b) There is a different approach to the complete data, by assuming that Y = T(Z) 
for some many-to-one map T : Z — JY. Then, Z is the complete data and Y is 
the incomplete data, but one does not identify missing data as such. 


The Bare-Bones EM Algorithm Fleshed Out 


Here, some of the details of the bare-bones EM algorithm are filled in by using 
explicit expressions for the conditional expectations. To that end, assume that one 
may take the dominating measure Rog to be Roo = Poo - Ooo, in the sense that 


Roo(A X B) = Poo(A) + Ooo(B) forall A € By and B € Bz. (26) 


Let (Y,Z) have density fyz (y,z|X0) with respect to the product measure 
Poo X Ooo. Then, for all A € Byxz and all measurable functions h on Y x Z 
with finite expectation E[h(Y, Z)], one may write 


"[A(Y, Z)] = i h(y.2) fy 2021) dPoo() dOo0(2) 


=[} [boa f.20-0)d Qua} aPoty) — ubini 
y 2 


“{\f h(y,2) ae a dOnt} Fy OX) dPoo(). 
yW 
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It is clear that this may be interpreted as the expected value of 


i n(Y, 2) SAGE dQ (2), 


and then interpret this in turn as E[h(Y, Z)| Y], the expected value of A(Y, Z) 
conditioned on Y. 
Now define the density of Z conditional on Y by 


fy 22x) 


27 
fF, 0h) oe 


fay ly.) = 


for those y for which fy (y|x) > 0 (and arbitrarily if (|x) = 0). Similarly, one 
defines 


hyz (y, z|x) 


28 
f(x) om 


Fyjz Ole. x) = 


for those z for which f, (z|x) > 0 (and arbitrarily if f, (z|x) = 0). So then Bayes’ 

rule yields 

Syz (y|z, x) te (z|x) 
AO) 


Say Gly. x) = (29) 


The conditional expectation of a measurable function (Y, Z) given Y is then 


HY ZI%.a] = [HW fay GY 2) deol. (30) 


Probabilists force us to add “almost surely” here. 
Now apply this to the conditional expectation of log fy. 2(Y, Z|x), with a guess 
x, of the true x. Then, 


olog fy z Y, Z|x)[¥, x1] 
= Eflog f, (Z|x) + log Az VIZ, NY, x1] 
-{ Ayyz Vz x) Fe Zl) 
dz fy (ix) 
Foy Wem) fe la) 
e), FCA 


log fz (zx) dQ 0(2) (31) 


log fy z(V lz. x) dQoo(2). 


For A,,(x|x1), this gives 
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An(xlx1) = [ G_ (2/1) log f, (21x) dQoo(2) + 


we yy Ayyz Mile x1) Fe a1) 


fy Wi |x1) log fz Wilz, x) dQo0(@), 32) 
y 1 


i=1 
with 


Pea) 
fy Milx1) 


1 n 
$20) = f@lm)-— D0 (33) 


i=l 


For the M-step of the algorithm, one has to minimize this over x, which is in 
general not trivial. The problem is simplified somewhat in the important case where 
fy . (y|z, x) is known and does not depend on x. Then, the problem reduces to 
solving 


minimize £,(x|x,) = -f gz (2x1) log fz (z|x) dQo0(z) 


(34) 
subjectto x eX. 
Note that 
Ly (x|x1) = KL(g, (Rafal s x)) + (terms not depending on x), (35) 


where KL(f, g) is the Kullback—Leibler divergence between the density f and g 
with respect to the same measure Qo, defined as 


f@ 


Kf.) = f | Flog | 2 x 


se ey x(a) dOvo(2). (36) 


Compare with (10). 

So, solving (34) amounts to computing what one may call the Kullback—Leibler 
projection of p, (- |x;) onto the parametric family P. If P is such that g,(- |x1) € 
P, then the projection is f,(- |x) = gz(- |x). 

The EM Algorithm Increases the Likelihood 
The expression for A,,(x|x1) (see (24)) may be reworked as follows. Using 


Fy 2% 1) = fey Gly) Fy Ol) 


(see (27)), one gets 
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i [log fy.z, Z|x) | x1, Y| = log f,, (|x) + Eflog Saty (Z|Y, x) |¥, x1], 
and so, 
A(x|x1) = La(X) + en (%1|x), (37) 


where L,,(x) is given by (17) and 


ental) © — Sf fay l¥ivw) log fey Gl¥iw) 400). 38) 


i=1 


It is now obvious that the EM algorithm decreases L,, (x): Let x2 be the minimizer 
of A(x|x,) over x € &X. Then, Ay(x2|x1) < An(x1)x1), and so Ly(x2) + 
€n(1|X2) S Ln (X1) + en (x1 |x1), oF 


Ln (1) — Ln (2) & en (21 [%2) — en (1 [X1) = Kn 1/2), (39) 


where 


n 


Kn (ulw) = > KL (fay 1%.) Fay 1%)) (40) 


i=1 


is a sum of Kullback—Leibler “distances”; see (36). Then, K,,(x|x2) > O unless 
xX, = X2, assuming that the conditional densities fz)y(- |Y¥,u) and fz\y(- |¥,w) 
are equal Qo, almost everywhere only if u = w. Then, the conclusion 


Ly (x1) > Ln (x2) unless x; = x2 (41) 


is justified. Thus, the EM algorithm decreases the likelihood. 

Unfortunately, x; = x2 being a fixed point of the EM iteration does not guarantee 
that then x; is a maximum likelihood estimator of x,. Equally unfortunately, even 
if one gets an infinite sequence of estimators, this does not imply that the sequence 
of estimators converges, nor that the likelihood converges to its maximum. Later 
on, the convergence of EM algorithms for special, convex maximum likelihood 
problems is discussed in detail. 


4 The EM Algorithm in Simple Cases 


In this section, some simple cases of the EM algorithm are discussed, capturing 
some of the essential features of more complicated “real” examples of maximum 
likelihood estimation to be discussed later on. It turns out that the EM algorithms 
are the “same” in all but the last example (regarding a finite mixture of unknown 


EM Algorithms 317 


densities), even though the settings appear to be quite different. However, even in 
the last case, the “same” EM algorithm arises via the empirical Bayes approach. 

A word on notation: The scaled negative log-likelihood for each problem is 
always denoted as L,,; L,,(x) in the discrete case, L,(/) in the continuous case. The 
negative log-likelihood in the M-step of the EM algorithm is denoted by A(x|x1) or 
variations thereof. 


Mixtures of Known Densities 


Let d > 1 bean integer and consider the statistical space (V, B, P), with Y = R¢’,B 
the o-algebra of Borel subsets of Y and P the collection of probability measures that 
are absolutely continuous with respect to Lebesgue measure. Consider the random 
variable Y with values in Y and with density 


fy) => xo(/aj(y), yey, (42) 
j=l 
where 4 1,42,...,@m are known densities and x) = (%o,1,%0.2,---;Xom)' iS a 
probability vector, i.e., x» € Vin, 
Vin = 4x € R” | x => 0 (componentwise), a x(j)=1>. (43) 
j=l 
Suppose that one has a random sample Y), Y2,..., Y;, of the random variable Y . 


The maximum likelihood problem for estimating f, (or estimating the probability 
vector X,) is then 


m 


zi 
minimize L,(x) = =F > log >. x(j)aj(%) 


i=l j=l 


(44) 
subjectto x € Vy. 
To derive an EM algorithm, missing data must be introduced. To see what could 


be missing, it is helpful to think of how one would simulate the random variable Y . 
First, draw the random variable J from the distribution 


A)=PV =fl=x), j =1,2,...,m. (45) 
Then, conditional on J = j, draw Y from the distribution with density a;. So, 
the missing data is J, a random variable with values in M = {1,2,...,m}. The 


associated statistical space is 


(MO Vin’ (46) 
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the o-algebra is the collection of all subsets of M, and the collection of all 
probability measures on M may be represented by V,,. Let a denote counting 
measure on M, i.e., for any A € 2M | 


a(A) =|A]| (the number of elements in A). (47) 


Then, it is easy to see that the distribution of (Y, J) is absolutely continuous with 
respect to the product measure fz x a, with density 


fy OD = AD fy OID = XG) 470), yeyof em, (48) 
Now, the complete data is (Y1, J1), (Yo, J2),...,(Yn, Jn) and the complete 
maximum likelihood problem for estimating x, is 
ae le : 
minimize — — 2 log {Way (x) subjectto x € Vy. (49) 
n T 
i=l 


Of course, the J; went unobserved, so one must compute the conditional 
expectations i [log {x(J) a, (Y)} | Y | . Now, 


I ( : ') oJ j 
Fy GIy) = + ~ = = (J) aj(y) . 
' ye ay(¥) Xo(J) 
p=1 


but of course, x, is unknown; approximate it by some initial guess xl e€ V,,, 
e.g., oe = 1/m for all 7. Then, the conditional expectation in question is 


approximated by 


[log (x(a, (¥)} |¥, x] = — 8 log {x(j)a;(¥)} x, ¥) dal) 


=— DF xPIG.Y) log {x aj)}. 


jEeM 
Wepya(v 
i x2l(j,¥) = x (j)aj(Y) . eM 
Ye ap) x"(p) 


p=1 
Then, the E-step of the EM algorithm leads to the problem 


minimize — s xPl(7) log {x(j) aij} subjectto x € V,, (50) 
jEM 
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where a;; = a;(¥;) for alli, 7 and xPl(j) = a ei xPl(7, ¥;), or 


n 


1 ij 
Plz) = xl). _ : 51 
x¥I(7) ON 2 - (51) 
De ain x"(p) 
p=l 


Taking into account that 
log {x(j) aj (Y)} = log x(j) + (a term not depending on x), 
one then arrives at the problem 
minimize A, (x|x?l) = — > xPl(j) log x(j) 
j=l (52) 


subjectto x € Vy. 
This is the E-step of the first iteration of the EM algorithm. Now consider the identity 
A (x|xPl) — A (xP) xP) = KL (x?) x), (53) 


where for u, w nonnegative vectors in R’, 


m 


KL) © YO Juli) tog ME 4 wi —wch. (54) 


j=l 


This is the finite-dimensional Kullback—Leibler divergence between the nonnegative 
vectors u and w. Note that the summand is nonnegative and so is minimal when 
u = w. So, the solution of (52) is precisely x = x1, This would be the M-step of 
the first iteration of the EM algorithm. The EM-step is then (51). 


A Deconvolution Problem 


The setting is the statistical space (R¢, B, P), where B is the o-algebra of Borel 
subsets of R¢ and P is the collection of all probability density functions on R@ 
(with respect to Lebesgue measure). Denote Lebesgue measure by jz. Let Y be a 
random variable with values in R? and with density Sy (with respect to Lebesgue 
measure); the interest is in estimating /\,. Now, assume that one is unable to observe 
Y directly but that instead one only observes a corrupted version, viz.,.W = Y+ Z, 
where Z is another R?-valued random variable. Assume that the distribution of Z 
is completely known; denote its density by k. The density of W is then K f,,, where 
the integral operator K : L' (R“, dj) > L! (R%, dy) is defined as 
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No) =f kor—») fOYdHO), weR’. (59) 


Note that k(- — y) is the density of W conditioned on Y = y, ie., fyjy(wly) = 
k(w — y) for all w and y, and then 


fyy wy) =kw-y) fy), wey eR. (56) 


So, assume that one has a random sample W,, W2,..., W, of the random variable 
W. The maximum likelihood problem for estimating fy is then 


minimize L,(f) = -- > log[kK f](W;) (57) 
i=l 


subjectto f EP. 


Recall that P is the collection of all pdfs in L'(R¢,d). It is impossible to 
guarantee that this problem has a solution. In fact, regularization is required; see 
section “Smoothed EM Algorithms.” Nevertheless, one can use EM algorithms to 
construct useful approximations to the density f,, by the expedient of stopping the 
iteration “early.” 

Note that one could view (57) as a continuous mixture problem, since Kf is a 
continuous mixture of known densities to wit the known densities k,(w) = k(w—y), 
y € R@, and the continuous weights are the unknown fy (y), y € R¢. However, the 
present approach is somewhat different. 

To derive an EM algorithm, one must decide on the missing data. It seems 
obvious that the missing data is Y itself or Z (or both), but the choice Y seems 
the most convenient. Thus, assume that one has available the random sample 
(W,, V1), (Wo, Y2),..., (Wn, Yn) of the random variable (W, Y). In view of (56), 
the maximum likelihood problem for estimating f, is then 


minimize A,(f) = 


1 n 
—— > log {k(W; — Yi) f(W; 
7D, HoetkH — 1) SH) es 


subjectto feEP. 
Since the Y; are not really observed, one must compute or approximate the 


conditional expectation E [log {k(W -—Y) f(Y)} | Ww]. Since the density of Y 
conditioned on W may be written as 


kw-y)frQ) 


Friw vl) = —Khlw 


then, approximating f, by some initial guess /,, the conditional expectation is 
approximated by 
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/ kKW-y) fi) 
Rd 


d 
KF) log f(y) du(y), 


apart from a term not depending on /. So then, the problem (58) is approximated 
by 


minimize -{ £0) log f(y) du(y) subjectto f EP, (59) 
Rd 
where 
1 kW-y) d 
= ; ) ; R*. 60 
AO) AO) yn aa IK AM) VE (60) 


This is the E-step of the EM algorithm 
For the M-step, i.e., actually solving (59), note that 


An(fIA,) — An(A IA) = ELA. SA), (61) 


which is minimal for f,. Thus, the solution of (59) is f = f, as well. Thus, the EM 
algorithm takes the form (60) iteratively applied. 


The discretized EM algorithm: The EM algorithm (60) cannot be implemented 
as is, but it certainly may be discretized. However, it is more straightforward to 
discretize the maximum likelihood problem (57). 

A reasonable way to discretize the maximum likelihood problem (57) is to restrict 
the minimization to step functions on a suitable partition of the space. Suppose that 
the compact set C, C R¢ contains the support of fy, and let {C; y=, be a partition 
of C,. Define the (step) functions a; by 


a= (Ce I@e CG), £=1,252 a, (62) 
where for any set A, the indicator function 1(y € A) is defined as 
I(y¢€A)=1 if yeA and =0 otherwise. (63) 
Then, define P,, to be the set of pdfs in the linear span of the a, 
Pm = 4 >. xj aj(-) |x € Vin (64) 
j=l 


Note that the a; are pdfs, and in fact, one could take the a;, 7 = 1,2,...,m to be 
any collection of pdfs. 
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Now, consider the restricted maximum likelihood problem 


1 n 
minimize — — ~ log[K f](W;) subjectto ff € Py (65) 
n 


i=l 
and observe that it may obviously be rewritten as 


- ; 1 n m ; 
minimize — = dX log dX aij Xj subject to x € Vn, (66) 
i= j= 


where fori = 1,2,...,n and j = 1,2,...,m, 


ay = [KOH — ray) du 


And this is all there is to it: The problem (66) is just a finite mixture problem with 
known distributions! Thus, the EM algorithm is as in section “Mixtures of Known 
Densities,” 


n 


k+y ml dij 
xX: — nee 3 a a See 67 
x ; 3 j (67) 


j j = i 
i= 
) k 
aip x5 
p=1 


with the estimator for f, induced by the representation of (64). 


Another EM algorithm? There is of course another way to derive an EM algorithm 
for the problem (65), viz., by introducing the missing data Y; as before. As for the 
unrestricted maximum likelihood problem (57), the E-step of the first iteration of 
the EM algorithm leads to the problem, analogous to (59), 


minimize -| L(y) log f(y) du(y) subjectto  f € Pn, (68) 
Ra 


with f, given by (60). Now, using the representations 


m 


fO=>o xa), fe) = >> x! a;0) 


j=l j=l 


with the step functions a;, fork = 1,2, the objective function in (65) may be written 
as 


aa leas d 
Yi toes iP fr) duo). 
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and of course 


_ att, DS dij cet [2] 
I fv) du(y) = xp + = d KAM (69) 


where 
ay =f KOK vai) day) =16) 7" f kOK —») dal. 
) 
Note that 


[FNS teas PSU, 


j=l 
Thus, the problem (68) is equivalent to 


m m 
minimize — » a logx; subjectto x20, > xj =l. (70) 
j=l j=l 


But it was already shown in section “Mixtures of Known Densities” that the solution 
is x = xl. So, the iterative step is exactly the same as in (67). As an aside, this 
is a case where the introduction of “different” missing data leads to the same EM 
algorithm. 


The Deconvolution Problem with Binning 


Consider again the deconvolution problem of section “A Deconvolution Problem,” 
but now with the extra twist that the data is binned. 

Recall that the random variable of interest is Y which lives in the statistical space 
(VY, By, P) with Y = R¢, By the o-algebra of Borel subsets of VY, and P the 
collection of probability measures on By, that are absolutely continuous with respect 
to Lebesgue measure. The density of Y is denoted by /,. The random variable Y 
was not observable. Instead, one can observe the random variable 


W=Y+4+Z, 


where Z is another random variable living in (VY, 6y,P), with known density 
denoted by k and independent of Y. Actually, with binned data, W is not observed 
either. Let £ € N and let {B; a C By bea partition of Y (or of a set containing 
the support of W). What one does observe is which “bin” B; the observation W 
belongs to. That is, one observes the random variable J, with J = j if 
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1(W ¢ Bj) =1, (71) 


cf. (63). Then, the statistical space of interest is (M, QM Vin); see (46). Of course, 
V,, is dominated by the counting measure, denoted by a; see (47). The density of J 
then satisfies 


fH) = ANB) = fA IO) dulw), j eM. (72) 


So, now one observes the random sample Ji, J2,..., J, of the random vari- 
able J, and the goal is to estimate f,. The maximum likelihood problem is then 


1 n 
minimize — — > log[K f|(By,) subjectto feEP, (73) 
n 


i=l 
which is equivalent to 


t 
1 
minimize — — ~ N; log|k f](B;) subjectto feP. (74) 
n 


j=l 


Here, the N; are the bin counts 


N= DoW), jem. (75) 


Remark 2. Later, for any function h : M — R, the more general identity 


Yo AG) = D5 Nj AG) 


i=l j=l 
will be useful. 


So, the real data are the bin counts, but it is advantageous to keep the J;. It has 
to be seen whether one can get away it, though. So, the starting point is (73) and 
not (74). 

To derive an EM algorithm, the missing data must be considered. It seems obvious 
that the W; are missing, and the treatment in section “A Deconvolution Problem” 
suggests that the Y; are missing as well. So, the complete data is the random sample 
(J;,W;, Y;) of the random variable (J, W, Y). This random variable lives in the 
statistical space (M x Y x Y, 6, Q), with B the o-algebra generated by the sets 
Ax BxC with A Cc M and B, C ¢€ By. Finally, Q is the collection of probability 
measures on B that are absolutely continuous with respect to the product measure 
a X pL X ww. The density of (J, W, Y) is then 
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Siwy Gwiy) =kw—y) fy (vy) we Bj), jeM,wryey. (76) 


The complete maximum likelihood problem for estimating f,, is now 
1 n 
minimize — — > log {k(W; — Y;) f(%)}  subjectto feEP. (77) 
n 
i=l 


This is the same as the problem (58). However, here one conditions differently. At 
issue is the conditional expectation E [log{k(W -Y)f()} | J | . Observe that 


Siwy Gwy) — k(w—y) fy (v) Ww € B;) 


VM)= = ; 78 
fuss DEG KF, MB) “ 
so that, replacing f, by some initial guess /|, one finds that 
S[log{k(W-Y) fY)}| J. Al 
k(w— y) fil) lw € By) 
= log{k(w — d d ; 
[Weather 9) Fo) 0) duty) 
Now, using 
log{k(w — y) f(y)} = log{ f(y)} + (a term not depending on /), 
one atrives at 
S[logtk(W—Y) FY} | J, fi 
AQ)ki) d 
» IAG) logt f(y) du(y) + rem, (79) 
where “rem” involves terms not depending on f, and 
kj) =f kov—y)duw), j eM. (80) 
Bj 
Note that then 
AB) = fk) HO AHO». G1) 


So, the E-step of the EM algorithm leads to the problem 


minimize -| fay) log f(y) du(y)_ subjectto =f EP, (82) 
Y 
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where (one would say: as always) 


" k,0) 
fly) = fiQ) ¢ = K ANB)" yey. (83) 
Since 
[ fie ano = > [kor F0y4n0) /ecriep} = 1, 


the solution of (82) is f5. So, the iterative step of the EM algorithm is given by (83). 
Actually, the situation is a little sticky, since (83) involves the J;. However, one may 
collect the J; with equal values, so that then 


n 


Nj = DU 1G = 4), 


i=l 
and so an equivalent definition of fo is 
m 


7 1 Nj kj) 
Ay) = AY) 7 2X [K fil(By) 


yey. (84) 


The EM algorithm is then obtained by iterative applying of the EM-step (84). 


Discretizing the EM algorithm: Note that a discretized EM algorithm may be 
derived by restricting the minimization in (74) to step functions on a partition 
{Cj 2 , Of a set containing the support of Y. With P,, as in (64), the restricted 
maximum likelihood problem with binned data is 


l 
1 
minimize — — > N; logi|k f\(C;) subjectto f € Pm. (85) 
n 


j=l 


One then derives the EM algorithm as in section “A Deconvolution Problem,” 
leading to 


£ 
Np ki 
let) ll >> ep FH12, ok (86) 


oe Soa x {kl 
dig Xq 
q=1 


where for p = 1,2,...,mandq = 1,2,...,£, 


EM Algorithms 327 


ane = | J kor» duoy ducw 
Cp Bg 
The estimators for fy are then 
£ 
k = 
A= xMic, We), yey. (87) 


j=l 


Finite Mixtures of Unknown Distributions 


The final simple case to be discussed is that of a mixture with a small number of 
densities belonging to some parametric family. 
Consider a random variable Y in a statistical space (YV, By, P), with 


P={f(- |x): x EX}, (88) 


a family of probability measures indexed by the (low-dimensional) parameter x € 
X. Assume that P is dominated by some measure Poo and that 


dP(- |x) 


| (y) = fy Ox), yey. (89) 


So, let Y be a random variable with density 


m 


LO= Yo wo DA (yx), y ed. (90) 
j=l 
Here, Wo = (Wo,,Wo2;--++>Wom) © Vin, the space of probability vectors (see (43)), 
and X) = (X%o1,%02.-++>Xom) € “Xm, thus defining %,,. (The notations x, ; and 
Xo(j) are used interchangeably.) 
Given a random sample Yj, Y2,..., Yj), the maximum likelihood problem for 


estimating w, and x, is then 


m 


1 n 
. . . ae ] . Y; , 
minimize ; = og a Wj fy( |x;) 


i=l j=l 


(91) 


subjectto weV,, x € Xn. 


To derive an EM algorithm, one must introduce the missing data. As in sec- 
tion “Mixtures of Known Densities,” the random index J € M = {1,2,...,m} 
would be a useful information because Foy Oly) = fy (y|Xo,;) - 
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So considering (Y1, J1), (Yo, J2),...,(Vn, Jn) to be the complete data, the 
maximum likelihood problem is 


1 n 
minimize ; d og{w( Ay ( |x(Ji))} (02) 


subjectto W EV, x € Xp. 


Similar to the development in section “Mixtures of Known Densities,’ now with 
initial guesses w!!! and x!", one obtains that 


t[logtw(J) fy (lx ))3 | ¥. wll, x! | 


wii) fy (Vx) 
-> y 


- logtw(i) fy (YIxG))}. 
jEemM s- wh) fy (¥ |x"(7)) 


p=1 


It follows that 


1 n 
| 3 D> logtw(Ji) fy Kile.) FY Fal 


i=l 


=— D> wl({) logw(j) + Ln! w"), (93) 
JEM 
where 
n Y; (lly; 
wei) = wij) - 2 > Mee . jeM, (94) 
n * 
=! (1 y, |xlll 
Yo wp) Ay (Yilx"(p)) 
pEM 
and 


Ay ila) log fy Wile) 


(95) 


Ly (x|xl, will) = = 2 


LS wiry fy (vile!) 


pEeM 


This is essentially the E-step of the EM algorithm. Note that the definition of w"! is 
in the by-now-familiar form. For the M-step, one must solve 
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minimize — > w?l(j) logw(j) + Ln (x[x!4], wll) 
jeM (96) 


subjectto we Vn, X € Xn, 


and this nicely separates. The minimization over w gives w = w"! as always, and 
x = xl is the solution of 


minimize <, (x|xl, wl!) subject to x € Xn. (97) 


Unfortunately, in general, there is no closed form solution of this problem. 
There are numerous examples of this type of mixture problems. See, e.g., [58,79]. 


Empirical Bayes Estimation 


There is another way of deriving EM algorithms for the mixture problem under 
consideration. Of course, that means one must introduce a different collection of 
missing data. The following development is from Eggermont and LaRiccia [39]. 
Starting from the beginning, consider the random variable Y with density 
Fv(- |Xo) with respect to Poo. Given a random sample Y;, Y2,..., Y, of the random 
variable Y , one wishes to estimate x,. The maximum likelihood problem is 


1 n 
minimize — — ~ log fy (Y;|x) subjectto xeEX. (98) 
n 


i=1 


So, what is the missing data in this case? It was already alluded to: the missing 
information is x,! In the so-called empirical Bayes approach, one considers x, to 
be a random variable in the statistical space (V,Bx,7), with 7 a collection of 
probability measures on 6, dominated by some measure 759. The complete data is 
the random sample (Y,, X1), (Y2, X2),..., (Yn, Xn) of the random variable (Y, X), 
with density 


Fy x) = fy OC) fyyy VP) = Ly) Ay Ol). (99) 


So, fx is the marginal density of X, but instead of prescribing a prior distribution 
on X, one’s task is to estimate this distribution without using prior information. The 
estimator of fy will tell us whether one parameter X = x, suffices for all Y;, viz., 
if the estimator of fy has most of its mass near x = x, or if one has (mostly) a 
mixture with a few components, or indeed a continuous mixture. 

Note that 


£0) = i: Fy (10) fy (8) d Too). (100) 
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Defining the integral operator K by 


IKNOY= [| LOWS AT, yey, (01) 


the maximum likelihood problem for estimating fy is 


1 n 
minimize — — Yo log(K f\(¥i) subjectto f eT. (102) 


i=1 


Note that the problem (102) is very much like the problem (57). Indeed, in very 
much the same way as in section “A Deconvolution Problem,” one derives the EM 
algorithm 


is LO) 


Stix) = fie(x) - re IK fal) , 
i=l ; 


eX, (103) 


This pretty much exhausts the simple examples that lead to this “same old” EM 
algorithm. 


5 Emission Tomography 
Flavors of Emission Tomography 


There are at least three flavors of emission tomography, viz., single photon emission 
tomography or SPECT (the C stands for “computerized’’), positron emission 
tomography or PET, and time-of-flight PET (TOFPET). In all of these cases, given 
measurements of the emissions, the objective is to reconstruct the three-dimensional 
distribution of a radiopharmaceutical compound in the brain, giving insight into the 
metabolism in general and blood flow, in particular, in the brain. 

In the single photon version, single photons are emitted in random locations in 
the brain and are detected (or not, as the case may be) by detectors situated around 
the head. In the positron version, single positrons are emitted in random locations. 
The positrons travel short distances until they are annihilated by single electrons, at 
which instances pairs of photons are created which fly off in nearly opposite random 
directions. The pairs of photons may then be detected by pairs of detectors. Thus, 
positron emission tomography amounts to double photon emission tomography. 
In the time-of-flight version of PET, the arrival times of the pairs of photons are 
recorded as well, which gives some information on the location of the emission. 
The time-of-flight version will not be considered further. Although the specifics are 
different, in their idealized form, the reconstruction problems for SPECT and PET 
are just about the same. For some of the details of the not-so-ideal circumstances in 
actual practice, see, e.g., [55,97]. 
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The Emission Tomography Experiment 


The data collection experiment for emission tomography may be described as 
follows. Consider a three-dimensional Poisson random field living in an open ball 
Q € R¢ (with d = 3). Here, “events” (viz., the creation of a single or double 
photon) happen with spatial intensity per unit time denoted by fz(z), z € Q. This 
means that for any Borel subset C of Q, the number N(C, ¢ , dt) of events that 
happen inside C during a time interval (t , f + 6¢) does not depend on ¢ and is a 
Poisson random variable with mean 


A[N(C, t ,6t)] = a [ Sz (2) dp(z). (104) 


Moreover, if C,C2,...,Cm are disjoint Borel subsets of Q and (¢;, t; + 64;), 
i = 1,2,...,m, denote arbitrary (deterministic) time intervals, then the counts 
N(Ci, t1, 6t1), N(C2, to, 6t2),..., N(Cm, tn, dtm), are independent. One may 
choose the unit of time AT in such a way that fz is the density of a probability 
measure with respect to Lebesgue measure on &2. Then, in a time interval 
(t,t +AAT), the number N = N(Q,t,AAT) of events that occur throughout 
Q is a Poisson random variable with mean 


af f,@dule=2. 
This may be written succinctly as 
N ~ Poisson(A). (105) 


For more on spatial Poisson processes, see, e.g., [24], section “The Shepp—Vardi 
EM Algorithm for PET.” 

Returning to the experiment, conditional on N = n, during the time interval 
(0,AAT), one collects a random sample Z), Z2,...,Z, (of sample size n) of 
the random variable Z, the random location of an event, with density fz with 
respect to Lebesgue measure. The random variable Z lives in the statistical space 
(Q, Be, Pe) with Be the o-algebra of Borel subsets of Q and Pg the collection 
of probability measures that are absolutely continuous with respect to Lebesgue 
measure. The events themselves are detected by detectors or pairs of detectors, 
denoted by B;, Bi,..., Bn, which one may view as disjoint subsets (or antipodal 
subsets) of a sphere surrounding Q. For each event at a location Z;, there is a 
random index J € M = {1,2,...,m}such that the event is detected by the detector 
(pair) B;. Thus, J lives in the statistical space (M, ge. Vin); see (46). The random 
variable (Z, J) is absolutely continuous with respect to the product measure ju x a, 
and its density is 


fz,@ I) = fz Gd £2, zeQ, jem, (106) 
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with f7|z determined by geometric considerations. See, e.g., [97]. Assume that this 
is known. Assuming that every event is detected, then f7)z is a conditional density, 
so 


m 


> fyzG) = 1 forall. (107) 


j=l 


Note that then 
f=] fyi f@due, FEM. (108) 


So, conditional on N = n, one may pretend to have a random sample (Z}, J), 
(Z2, J2),...,(Zn, Jn) of the random variable (Z, J). This gives rise to the usual 
form of the actual data to wit the bin counts 


n 


N; = >> 1; = J). (109) 


i=] 


Continuing (and no longer conditioning on N = n), then Nj, No,..., Nm are 
independent Poisson random variables with E[N;] = [K fz](/), 


Nj ~ Poisson(A[K f,](/)). ji €M, (110) 


where K : L'(Q, dw) —> L'(M, da) is defined by 


KAA =f fy2Ge@due. jem. an 


This concludes the description of the ideal emission tomography experiment. In 
reality, quite a few extra things need to be taken into account, such as the attenuation 
of photons by tissue and bone in the head; see, e.g., [55]. For the treatment of 
background noise, see, e.g., [42] and references therein. 


The Shepp-Vardi EM Algorithm for PET 


After these preparations, the maximum likelihood problem for estimating fz may 
be formulated. The observed data are the count data N;, No,..., Nm, which leads 
to the problem 


1 m 
minimize — W > N; logik f\(j) subjectto fe Po. (112) 
j=l 
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Alternatively, and this is actually more convenient, one may view the total number 


of detected events N and Jj, Jj,..., Jy as the actual data, which gives 
N 
oe 1 : 
minimize — W ~ logiK f|(Ji) subjectto fe Po. (113) 


i=1 


In view of Remark 2 following Remark (75), these two problems are equivalent. 

So far, nothing has been said about approximating/representing the estimators in 
terms of pixels or voxels. Let {C Ahem be a partition of Q, and let Pe C Pg be 
the space of step functions that are constant on each C,. Thus, with V; defined as 
in (43), 


L 
P,=1>- xp bo(-) [xe Ver, (114) 
p= 
where bz) =|C, ("Wee c,), 2em, 


(Note however that one may take other basis functions.) The discretized maximum 
likelihood problem is then obtained by restriction 


shee LY 
minimize — W > logik f](Ji) subjectto fe P,. (115) 


i=1 


From the description of the experiment, it is clear that the missing data are the 
Z;, so the complete data set is (Z;, J)), (Z2, J2),...,(Zw, Jn), a random sample 
(of random sample size N) of the random variable (Z, J). The joint density of 
N, (Zi, Ji), (Z2, Jo), bide (Zn, Jy) is then 


n 


Xr n 
e* UT file) Ze), (116) 


nN: 


so that the complete maximum likelihood problem is 


N 
ae 1 : 
minimize — WV y log { fy, (J;|Z;) Zi subjectto f €P,). 


= (117) 
In the objective function, the terms corresponding to the Poisson distribution of N 
have been omitted, and the scaling 1/N was applied. 
For the E-step of the EM algorithm, consider the computation of 


[log {fz il) SZ} | I. Ai]. (118) 
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assuming the approximation /; to fz. Since 


Fiza fz @ 


(z|j) = ; : (119) 
Pais lj LP (/) 
this gives for the conditional expectation (118) the expression 
fyzJla f,@ 
— [| AE hog FO dn, (120) 
Q Ti (j) 


where the contribution involving the known f7z may be ignored, since it does not 
depend on f/f. 
Consequently, the E-step leads to the problem 


minimize -{ A(z) log f(z) du(z) subjectto feP,, (121) 
Q 
where 
N 
1 Fg Vile 
LO=A@)= - ae (122) 
(fi tyes) fs) dus) ) 
Note that / is a density. 
In terms of the representation of elements in P,, 
e 
{@=)> x4), (123) 
p=l 


with a,(z) =|C,|~! 1(z € C,) as in (62), and likewise for f; and fy, this leads to 


£ 
[ fald) log f(2) dul) = D> x2) tog {xp Cp} 


p=1 


U l 
a > xl log x, —log|C> | >» xl (124) 
p=i1 p=l 


e 
= ~, x? log x, —log|C, |, 
p=1 


EM Algorithms 335 


where 
i= a(Ji, P) 
x? = xl . W dX r , (125) 
7 > a(Ji, Pp) xi 
q=1 
with 
aj. p) = i fg Gla, © du = ; fyzlilddu@. (126) 


va 
Note that ~ xl = 1 and, by (107), that 
p=1 


m 


Y> a(j,p) =1 forall p. (127) 
j=l 
So, the E-step gives 
£ 
minimize — > xl logx, subjectto xeV,. (128) 


p=1 


In section “Mixtures of Known Densities,” it was already shown that the solution 
of the problem (128) is given by x = xl. So (125) is the iterative step of the EM 
algorithm. Of course, using Remark 2, the iterative step for x”! may be rewritten in 
terms of the bin counts as 


I se Nj aj, p) 
j= [< 
Y= a(j, p) xt! 


q=l 


2] _ +L] 
xy =X 


(129) 


Observe again the similarity with the EM algorithms for the simple examples in 
Sect. 4. 


Remark 3. The problem (115) is not really discretized. The actual discretized 
problem is 
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£ 


1 m 
minimize — W > N; log[Ax]; + Xp subjectto xeEV, (130) 


@? 
j=l p=1 


with V, given by (43), and A € R”** has components a(j, p) given by (126). This 
uses Remark 2. 


Remark 4. To finish, note that the original derivation by Shepp and Vardi [88] 
involved the missing data M(j, p), the number of events in each “cell” of Q that 
contribute to the counts N,, 


m 


MG p) =>) WG =/UZEC,), p=1,2,...,6. 


i=! 


This calls for a rather complicated relation between the M(j, p) and the N;. In 
particular, one does not have random samples of the appropriate random variables. 
It gets much simplified if one introduces the random variables J,, In,..., Im which 
indicate to what “cell” the event Z; belongs to. So, J; = p if 


\(Z; € Cy) = 1, 


Then, for the complete data, one gets back to considering random samples of 
random variables, viz., (J, J). This would provide for an alternative approach to 
discretization but would lead to the same EM algorithm. This is essentially the “list 
mode” approach of Parra and Barrett [77]. See also [10]. 


Prehistory of the Shepp-Vardi EM Algorithm 


The earliest reference to maximum likelihood estimation in emission tomography is 
the aforementioned paper by Rockmore and Macovski [82]. In astronomy, an early 
reference is Lucy [70]. The EM algorithm for these maximum likelihood problems 
was introduced by Shepp and Vardi [88] and independently by Lange and Carson 
[64]. See also [20] for a completely different setting. The EM algorithm for SPECT 
is essentially the same; see, e.g., [60] and references therein. 

The algorithm itself may be viewed as a method for approximately solving the 
integral equation with moment discretization 


KAD =. J =1,2,...,m, (131) 
with K as in (111). In particular, this may be applied to Fredholm integral equations 
of the first kind. As such it was independently discovered in various settings many 
times over, by Tarasko [93] and Kondor [61] in Physics, by Richardson [81] and 
Lucy [70] in Astronomy, and perhaps other authors. It is interesting to note that 
both Richardson [81] and Lucy [70] derive the algorithm based on probabilistic 
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considerations involving Bayes’ theorem, as in (119). For more on the integral 
equations aspect, see also [74]. 


6 Electron Microscopy 


In this section, a recent application of EM algorithms at the bleeding edge of science 
is considered. As far as the EM algorithm is concerned, the foundation is far from 
complete, whether it be practical or theoretical. 


Imaging Macromolecular Assemblies 


Structural biologists are interested in the shape of biological objects at the macro- 
molecular level. The tail of the T4 bacteriophage is a famous example. Such objects 
are referred to as macromolecular assemblies. To view objects that are this tiny, 
electron microscopy seems to be the only tool available. Its use in structural biology 
goes back to DeRosier and Klug; see [21]. Ideally, one would like to take a single tail 
of the T4 bacteriophage, say, obtain electron micrographs (projections) from many 
directions, and reconstruct the three-dimensional structure of the tail. Unfortunately, 
the bombardment with electrons destroys the object, so that only one projection can 
be taken. The biologists have found a way around this, but it comes at a price. 
Very roughly speaking, many tails are isolated and suspended in a thin layer of 
water, which is then rapidly cooled to below freezing. This results in vitreous water, 
with the tails suspended in it but randomly located and oriented. A single electron 
micrograph of this layer is then taken. This is equivalent to taking projections of a 
single tail in many different directions, corresponding to the random orientations. 
For a precise description and analysis of the procedure, see [44]. Now, the price 
one pays is that one has many projections of the tail but in random unknown 
directions. Since these random directions may be viewed as missing data, it is clear 
that EM algorithms may be used. This was first realized by Scheres et al. [85]. A 
complication is that the objects can appear in conformational states, which means 
that one has a mixture of finitely many (different) objects. Another complication is 
that the signal-to-noise ratio is typically quite small, in the 10 % range. 


The Maximum Likelihood Problem 


Mathematically, following Scheres et al. [83,84], the setup may be described as 
follows. Each object in the thin layer may be considered as being randomly chosen 
from a finite collection x?,x5,...,x? of « objects. Its position and orientation in 
the thin layer is described by five real-valued parameters: two location parameters 
and three Eulerian angles describing its orientation. Denote them by © and the set 
of all possible © by &. The problem of finding the location parameters is referred 
to as the problem of alignment. For low signal-to-noise ratios, maximum likelihood 
methods seem to be preferable [89]. 
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So, for a random object, one observes the projection in the form of a discretized 
image Y, 


Y=C#eR xe +e. (132) 


Here, K is the random index into the collection of possible objects, Ryx, is 
the projection data of the object in the “direction” ©, C is the (known) contrast 
transfer function (due to the experimental setup), and * denotes the two-dimensional 
discretized convolution operation. Finally, ¢ is the noise, assumed to be normal and 
isotropic, i.e., the components of ¢ are jointly normal and the components of the 
variance—covariance matrix V, satisfy 


BEng Ers] = Up—rg—s> (133) 


where © p—;g—s is a function of (p—r)? +(q—s)’, the (squared) Euclidean distance, 
only. In terms of the two-dimensional discrete Fourier transform, this means that 
(ignoring boundary effects) 


ae aan 2 
[eee enel= [o7],. for P=R and Q=S, (134) 


and = 0 otherwise. Moreover, on 0 is rotationally symmetric, i.e., it is a function 


of P? + Q?. Unfortunately, Gs g is unknown; it must be estimated. Moreover, a? 


varies with Y. 
So, the distribution of Y conditional on K = k and © = @ is given by 


1 oO 
exp (= IC Ry af, ) 


0,x7) = : (135) 
Frie.x (y| t) (22)N/? det(V,) 
where N is the size of Y (or y) and ale = z’V-~!z. In terms of Fourier 
transforms, this reads as 
2 
Oo a —2 A oj’ _ fs 
IC + Rost, =D [Ing [LC], olny no | 
we (136) 
log det(V,) = ye log [ol], 58 
P.O 


Now introduce the state of the system Y one wishes to estimate and the initial 
state of one’s understanding of the system, 


cS= \w°, eae, and SU = {bara oll, xf) ol, (137) 
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where w, = P[K = k], fojx is the density of © conditional on K, and x° and o, 
are as before. The current understanding of the system is comprised of one’s best 
guesses so far for the true system. 

The distribution of Y may be expressed as 


F,0) =F | Fe 118.) Foy 1K) A008), (138) 
k=1 = 


where v = {4 X w, with jz the Lebesgue measure on R? and w the surface measure 
on the sphere in R?. Since it is reasonable to assume that the random location 
parameters are independent of the random orientation, then 


Fo (91, 92, 3, 4, 5) = g(A1, 2) h(O3, 4, 5), (139) 


for appropriate densities g and h. This reduces the actual dimension of the problem, 
but for notational ease, such a specialization will not be made. 

Given a random sample Y,, Y2,..., Y,, of Y, the maximum likelihood problem 
for estimating the unknown objects x), x2,...,X, may then be formulated: 


minimize -= > log (>: Wk [ (0; |k) Fy ine Wilxe, ‘v0 (140) 
k=1 = 


i=1 


The unknowns are the probability vector w, the densities g(@|k) (keep (139) in 
mind), the unknown objects x;,X2,...,X,, and the variances [o7 | P.O" Note that 
there is some similarity with the empirical Bayes problem of section “Empirical 


Bayes Estimation.” 


The EM Algorithm, up to a Point 


Obviously, the goal is to derive an EM algorithm for the solution of (139), but the 
final algorithm is not quite the real thing. It is clear that the missing data for each 
observed projection Y; consists of the orientation, denoted by ©;, and which kind 
of object one is looking at, encoded in the index K;. The e and the objects x, are 
considered as parameters. So, the complete data set is (Y;,@;, K;),i = 1,2,...,n, 
and the complete maximum likelihood problem is then to minimize 


def 


1 n 
An(S) = —— ¥ log (x, POIK) Syjo,¢ Yili. %x,+9;)) (141) 


i=1 


over all probability vectors w, all densities y(- |k), all variance matrices a. and all 
X1,X2,...,X,. However, recall that the g( - |k) have a simple structure. 

For the E-step, the conditional expectation E[A,,(S) | Y] is needed. By Bayes’ 
rule, the distribution of (K, ©) conditional on Y is described by 
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Dy ox Ak) Fyjo.x 19, xx, 0) 
fy Y) 


P[K=K|Y=y] foyy ¢ @ly.28) = 


So, with the current state S!", and setting Y,, = {Y;, Y2,..., Y,}, one gets 


E[A,(S) |Yn, S!] = 
ss i gi (9.¥,) log (~, GOK) frre x (1,19, xf, 0/")) dv(0)(142) 
k=1 °° 


where 


ol go"! (6|k) aor (alee) 
AM) 


gt (0, ¥,) = , with (143) 


AO) =O al! f eMC Fra x (10M af!) ave). 44) 
k=1 7 


This completes the E-step. 
The M-step deals with the minimization of E[A,(S)|Y,,S"] over S$. This 
separates into three problems. First, estimating @ may be done by solving 


minimize — J) > (log Wk) i nila) dv(@) 
k=l = (145) 


subject to @ is a probability vector, 


where 


n 


1 
hy! (8) = — D0 gy (O1Y)). (146) 


i=1 


One verifies that the solution is 7 = wl, 
of! = / nla) dv(0),  k =1,2,...,«. (147) 


Second, estimating the g( - |k) may be done by solving 


minimize -y f h’(6, ¥;) log p(O|k) dv(0) 
k= °8 (148) 


subjectto (- |k)isapdf, k = 1,2,...,kK. 
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This separates into « minimization problems for the g( - |k). One verifies that the 
solutions are fork = 1,2,...,K, 


H) [1] 
1 n Frio K (Vile. xf » 0; ) 
[2] = oil : 
g(6|k) = g@@|k)-— YO (149) 
ni frig Hilk) 


where 


fil ilk) = i oMBIK) Frio. (Vil8 x¢).0)") dv(@). (150) 


Y|K 


Note that (147) and (148) are again multiplicative algorithms. 
Third and last, one must estimate fy|@ x, which boils down to estimating the x, 
and o;. The problem is to minimize 


K 1 n 
~ pS i Pi > ge (6, Yi) log fye.« Wil, Xx, 01) dv(9) (151) 
k=1 °° i=l 


over x; and 0;. Since log fy)4 x (Vil@, xx, ai) equals 


|O.K 


y [207] 


P,Q P.O 


here too the minimization problems separate. This may be solved for each x, by 
minimizing 


| Cr -[(Re x%)"Ip.9 - [%] 


[207] 


P,Q 


1 n 
wisi(x.) = f > > 86. ¥) 


i=1 


P,Q 


Denoting the minimizing x, by xl , then the new 0; is 0; = of with 


2] _< 1 py y 
le lea a 2) 7 22 EM) 


i=1 


Cra’ (Re ey] : . a 


x 


dv(@). 
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The Ill-Posed Weighted Least-Squares Problem 


Up to this point, the M-step has been carried out exactly. The last part is to minimize 
WLS; (xi). This is a weighted least-squares problem, which is nice, but it is ill-posed 
(or ill conditioned after discretization), which implies that one cannot and should not 
solve it exactly. In fact, Scheres et al. [84] employ a Wiener filter to stably implement 
the “exact” deconvolution procedure 


A [Yi] 
I(R a) = "2 forall P,Q, 
P.O Cro 


as in Penczek et al. [78], compared with Byrne and Fiddy [14], after which a 
weighted least-squares version of ART (WLSART) is applied. 

It should be observed that these problems are very large and are computationally 
very expensive. It is clear that there is much room for algorithmic development, but 
the present discussion ends here. 


7 Regularization in Emission Tomography 
The Need for Regularization 


It is clear that the simple deconvolution problem (57) and the more complicated PET 
problem (112) are ill-posed, let alone the electron microscopy problem of Sect. 6. 
As already observed, in the PET problem (112), one is trying to solve the compact 
operator equation with moment discretization 


IKfIG)=6;, 7 =1,2,....m, (152) 


where b; = N;/N. The possible nonexistence of solutions is dealt with by 
considering the maximum likelihood problem, which may be reformulated as 


minimize KL(b,&f) subjectto feP, (153) 
where KL is the discrete Kullback—Leibler divergence (see (54)) and 


Rf = (KIA). [Kf](2),.... [Kf ](m))”. 


The problem (153) is similar to a least-squares problem. However, as in the case 
of the least-squares approach to compact operator equations, this still does not take 
care of the ill-posedness of the problem. So, the problem (153) must be regularized. 

The standard and practically the most often used method is to use the EM algo- 
rithm and stop the algorithm at some appropriate point in the iteration. See, e.g., [69] 
for practical aspects and [80] and [51] for some theoretical results. The alternative is 
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essentially Tikhonov regularization of the negative log-likelihood, which also comes 
in the guises of Bayesian or maximum a posteriori (MAP) likelihood estimation, 
Gibbs smoothing, or just roughness penalization. See [36, 46, 54, 63, 73]. A new 
twist is the use of total variation regularization and nonlinear diffusion filtering 
in connection with maximum likelihood estimation and EM algorithms (see, e.g., 
(3,6, 31,87, 100]), but unfortunately, this will not be discussed further. 

In many cases, the E-step of the EM algorithm may be carried out explicitly, 
but not so for the M-step. Here, some obvious modifications of the EM algorithm 
or extraneous iterative methods must be introduced. However, a few examples of 
explicit honest-to-goodness EM algorithms for regularized maximum likelihood 
problems are discussed: the NEMS modification of the EMS method of Silverman 
et al. [90] and two EM algorithms for Good’s roughness penalization. 


Smoothed EM Algorithms 


In this section, the discussion centers on the EMS algorithm of Silverman et al. [90] 
and the nonlinearly smoothed NEMS variant of Eggermont and LaRiccia [36] in the 
context of the deconvolution problem of section “A Deconvolution Problem.” Silver- 
man et al. [90] realized the necessity for regularization of the maximum likelihood 
problem in that the EM algorithm produces increasingly rougher estimators. Initially, 
this is good since one typically starts out with a uniform estimator and more features 
of the signal appear. However, as the iteration progresses, the estimator becomes 
increasingly nonsmooth, giving rise to spurious features. But Silverman et al. [90] 
figured they knew how to fix the nonsmoothness: Add a smoothing step to the EM 
algorithm. 
So, let S, be a smoothing operator in the form 


SAI) =f SK-aFOdua@. y eR (154) 


where S;(z) = h~4¢ S(h7'z) for some bounded, continuous, symmetric pdf S € 
L'(R®), possibly with compact support. The EMS algorithm then takes the form 


LO ky - 
Sk+1/2(2) = fx (2° fe » pean (155) 


i=] 


Seti = Sn feti/2- 


So the general step of the EMS algorithm is one step of the EM algorithm followed 
by one smoothing step. 

Silverman et al. [90] apply this algorithm to the simple problem of stereology 
(a integral equation on a compact interval) and to positron emission tomography. 
In both cases, it seems to work quite well. Of course, the question is whether 
the algorithm (155) converges and, if so, what it converges to. Regarding the first 
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question, see [65,99]. Characterizing the limit is not so easy, e.g., if one has a fixed 
point of the iteration, does one then have a point where the gradient of some log- 
likelihood-like function vanishes? 

In retrospect, it is clear that adding a smoothing step to the EM algorithm is a 
fundamentally sound idea, but the way it is implemented is not “right.” Indeed, in 
view of the multiplicative character of the EM algorithm, it seems that multiplicative 
smoothing is called for. So, with S;, as before but with S;,(z) = 0 everywhere, define 
the nonlinear smoothing operator V on nonnegative functions f by 


IN (AIO) = exp ((Sr(log fJ]Q),  y ER’. (156) 


Note that by convexity, [IV(f)](v) < [Si f](v), so that V(f) is always well 
defined. One verifies that NV’ performs multiplicative smoothing, i.e., 


N(f +8) =N(f)-N(g), (157) 


where the dot means pointwise multiplication: [f-g](v) = f(yv)g(y) for all y. It 
now turns out that the smoothed maximum likelihood problem 


minimize — a logi|KN(f)|(¥%i) subjectto fEeP (158) 


i=l 
admits the EM algorithm 
fetips = Nf), 


n 


Sk+2/3(2) = fe+1/3(2) - : DD 


i=l 


k(¥; — z) 
IK fietiy3](¥i) 


(159) 


Feta = Sh fr+2/3 


see [36]. In addition, the problem (158) has a solution and it is unique, and the 
algorithm (159) converges to this solution in the Kullback—Leibler sense. See 
section “Monotonicity of the Smoothed EM Algorithm.” 

The algorithm (159) is referred to as the NEMS algorithm: The general step 
consists of a nonlinear smoothing step, one step of the original EM algorithm, and 
a final (linear) smoothing step. The practical performance on the toy stereology 
problem is just about indistinguishable from the NEMS algorithm except that 
with the same smoothing operator S;,, the NEMS algorithm does about twice the 
smoothing of the EMS algorithm. Note that the question about the proper choice 
of the smoothing operator (or smoothing matrix in the discrete case) arises. This is 
in effect a question about the selection of the regularization parameter in ill-posed 
problems. Unfortunately, this is not addressed in this chapter. 
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Good’s Roughness Penalization 


Good’s roughness penalization of the deconvolution problem is a particular form of 
Tikhonov regularization. The roughness penalty function of Good [47] is 


“if IVf@P 
af) =; [ edn. (160) 


(The factor + is for convenience only.) The maximum penalized likelihood problem 
is then 


ns Y> logic / 1%) +f f(z) du(z) +h? ®(f) 
a= R¢ (161) 
subjectto feEP. 


One can now perform the E-step as in section “A Deconvolution Problem” to arrive 
at the problem 


minimize — / fy) log FO) duty) + i; fy) dy +P @(f) 
Ra Rd (162) 


subjectto f €P, 


where 


n 


1 k(W; — 
AO) =f0)- = > " aca yeR? (163) 


i=] 


At this stage, the change of variable u = al f is obviously(?) useful. The problem 
then becomes 


minimize —2 il f(y) log u(y) du(y) + ul? +2? || Vu |? 
Rd (164) 


subjectto u€ L?(R%), u> 0, 


where || - || denotes the L7(IR) norm. Here, it is convenient to drop the constraint 
|| u || = 1. Note that (164) is a convex minimization problem. The Euler equations 
are given by the boundary value problem 


ee) in R¢, 
u (165) 


Vu(y) —0 for |y|—o, 
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where u is nonnegative. The M-step amounts to solving the boundary value problem. 

The resulting algorithm converges, by arguments similar to those for the related 
discrete case of the next section. See section “Monotonicity for Exact Gibbs 
Smoothing.” 

For the positron emission tomography problem, Miller and Roysam [73] arrived 
at the analogue of this equation and solved the boundary value problem by finite 
differences, using Jacobi’s method on a massively parallel computer. Of course, 
other methods come to mind. 


Another EM algorithm: There is another way to proceed. With the change of 
variable u = asf as before, the objective function in (161) becomes 


1 n 
— = D7 logic w’y(%) + [a |? + 0? | Vel. (166) 
i=1 


Now, introduce the convolution operator S;, with kernel S;,(z) = h7! S(h7'2), 
defined via its Fourier transform as 


S(o) = / S(e 7%) du(z) = {1 + | 2x0 AA? (167) 
Rd 


for w € R¢. Here and below, | @| denotes the Euclidean norm of w, and (@, z) 
denotes the inner product on R. In fact, then 


SQ = 20-0? g- EMF | g|-O-Y? Kea-nyallz), zceR*, (168) 


where K,, is the modified Bessel function of the second kind of order v. Aronszajn 
and Smith [1] turn out to be the ideal reference for this. 
The convolution operator is defined as 


[Sn f1@ = t: Sie 5) fs) duals), ce RY, (169) 


and satisfies (S;f)*(@) = {1+ Qr2h|o pay? fo) for w € R¢. 
The net effect is that v = S;u satisfies 


| wl? +A? | Vel? = Iv, (170) 
so that the final change of variable f = 0t(w), where 
MOO) = [ivr], yer’, (171) 


transforms the original maximum likelihood problem (161) into 
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1 n 
inimi a log[KCIN Y;) + d 
minimize — 7 J logic amowy}0r) + fw) duc = 


subjectto weP. 


(Actually, the pdf constraints are treated a bit cavalierly. Obviously f and w cannot 
both be pdfs, but let it pass.) 

It now turns out that there is an EM algorithm for the smoothed maximum 
likelihood problem (172) to wit 


Wie41/3 = M,), 


y2 1 2 k(Y; — 2) 
Wie42/3 (2) = Pes} n dX Kwyp i310) 


We = {HQ}? Sy" y2,1@- 


(173) 


It has the same monotonicity properties as the NEMS algorithm; see section “Mono- 
tonicity of the Smoothed EM Algorithm.” The original method of Miller and 
Roysam [73] satisfies similar monotonicity properties (assuming that (165) is solved 
exactly). See section “Monotonicity for Exact Gibbs Smoothing.” 


Gibbs Smoothing 


Whereas Good’s roughness penalization was essentially aimed at the continuous 
setting, attention now turns to a purely discrete point of view. So, let us consider the 
discrete maximum penalized likelihood problem 


m £ 
minimize — a b; log[Ax]; + > Xp + AG(x) 
j=l p= 


(174) 
subjectto x € Vis 
with V, given by (43) and A € IR”** has components a(j, p) given by (127) 


and A > 0 is the regularization parameter. The typical form of the penalization 
associated with the name of Gibbs smoothing is 


G(x) = So Wyq 6 (07 (Xp — 4), (175) 
Pq 


for some convex function ¢ and nonnegative weights w,, and positive o. Some 
typical examples for ¢ are ¢(t) = logcosh(t) and @(t) = |t| for t € R. The 
nonzero weights w,, determine a neighborhood system. The neighborhood of the p- 
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th component of x is given by {¢ | Wpq > 0}. Although x was encoded as a column 
vector, one should think of x as a two-dimensional image or three-dimensional 
structure, so that neighboring image elements may have widely differing indices 
p and q. See [45, 46, 66]. The approach of (174) originated with Green [49]. 

The role of the penalty term is to penalize differences in neighboring components 
of x, but large differences are not penalized much more. In fact, this is an argument 
for choosing #(t ) = min(|¢|,6) for some 6. 

To solve the problem (174), again proceed iteratively, and perform the E-step of 
the EM algorithm. As before, this gives 


l r 
minimize — > xf logx, + > Xp +AG(x) 


p=1 p=1 


(176) 


subjectto x €V,, 


with X?! given by (129). For convenience, the constraint that x € V; is now dropped. 
To solve the resulting problem, set the gradient equal to 0, 


2] 
_ “P+ 1+AVG(x) =i (177) 
P 


Now, the one-step-late idea of Green [49] is to approximately solve this equation by 


22 
sal ae 
Po 1+ A[VGRN)], | 


p=1,2,...,2. (178) 


This is referred to as OSL-EM. Green [49] reports that this works well for small 1. 
Regarding its convergence under appropriate conditions, see [62]. If (177) is solved 
exactly, then the resulting algorithm has the usual nice monotonicity properties; see 
section “Monotonicity for Exact Gibbs Smoothing.” 

Hebert and Leahy [54] observed that (178) is similar in spirit to Jacobi’s method 
for solving systems of linear equations, and they noticed that the Gauss—Seidel 
analogue of sequentially solving (177) speeds up the computations. See also [41]. 
For other ways to accelerate EM algorithms, see Sect. 10. 


8 Convergence of EM Algorithms 


The convergence of the Shepp—Vardi EM algorithm is based on two rather remark- 
able monotonicity properties of the EM algorithm, established using analytical 
methods by Miilthei and Schorr [75]. Unfortunately, the geometric approach of 
Csiszar and Tusnady [23] that seems to explain why the Miilthei-Schorr approach 
works is not discussed. See [38]. However, the methods generalize in different ways. 
See Sect. 9. 
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The Two Monotonicity Properties 


Consider the discretized maximum likelihood problem of positron emission tomog- 
raphy, repeated here for convenience: 


m £ 
minimize L(x) = -\ b; log[Ax]; + bm Xp 
p=1 


j=l 
subjectto xe V,, (179) 


where b; = N;/N. Here, V, is given by (43) and A € R”** has nonnegative 
components a(j, p) given by (126), with unit column sums 


m 


Y> a(j.p) =1, P= 1; 2aca5h. (180) 


i=l 


It is clear that the problem (179) is convex and that solutions exist. The uniqueness 
is guaranteed only if A has full column rank. Regardless, the set of minimizers, 
denoted by C, is convex. 

Recall that the EM algorithm for solving (179) is, fork = 1,2,..., 


k+l] — lk) p47, | = 
tC Se Ae PSM 2y reek, 


by (181) 


ik] — =1,2,...,m. 


"? ~ Taxky? / 


starting from some initial strictly positive probability vector x!!. 
The two monotonicity properties are as follows: 


L(x) — LETH) > KLE TY, xl) > 0, (182) 
and, if x* is any solution of (179), 
KL(x*, x1) — KL(x*, x FY) & L(x!) — L(x*) = 0. (183) 


The meaning of the first monotonicity property is clear: It says that the likelihood 
decreases if successive iterates are different. The second one says that the iterates 
get closer to every minimizer as measured by the Kullback—Leibler “distance.” The 
everyday image is that if one thinks of the set of minimizers as an airport, then 
the iterates land like a helicopter, not like an airplane. This kind of monotonicity is 
called Féjér monotonicity. 

The two monotonicity properties imply that the EM algorithm converges. 
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Theorem 1. /f x"! is strictly positive, then the sequence {x"}, generated by the 
EM algorithm (181) converges to a solution, say x**, of the maximum likelihood 
problem (179). In particular, 


lim KL(x**, xl) = 0. 
k->oo 


Proof. The first inequality says that the negative log-likelihood is strictly decreas- 
ing, unless x1 = x+1 Tf xl = xl+" does indeed hold, then the second 
inequality says that L(x!) = L(x*), so that x! is a solution of (179). In 
general, the second inequality implies that {KL(x*, x])}, is a decreasing sequence. 
Since the sequence is bounded from below (by 0), it must have a limit, but then 
KL(x*, x1) — KL(x*, xt") —s 0, which implies that 


LG) —> LO"). (184) 


So, the negative log-likelihood converges. Finally, since > x = |, the sequence 
fxlhly p is bounded and hence has a convergent subsequence, say, with limit x**. 
By (184), then L(x**) = L(x*), so that x** is a minimizer also. Now, in the second 
monotonicity property, one may replace x* by x**, and then {KL(x**, x])}, is 
decreasing. Since a subsequence converges to 0, then the whole sequence converges 
to 0. (i 


It should be observed that the theorem is actually not very useful: When using the 
algorithm (181), one will always stop the algorithm well short of convergence. See, 
e.g., [69, 80]. Thus, the existence of maximum likelihood estimators is moot. One 
may think of this as an unfortunate side effect of discretization. For the continuous 
version, say, for the deconvolution problem, one does indeed have the analogues 
of the above two monotonicity properties, but the second one is vacuous, since the 
continuous maximum likelihood problem has no solutions. For the NEMS algorithm, 
one can show the existence of solutions as well as its convergence by way of 
the two monotonicity properties. See section “Monotonicity of the Smoothed EM 
Algorithm.” 

In the following subsections, the two monotonicity properties are proved for 
the standard discrete Shepp—Vardi EM algorithm, for the continuous version of the 
NEMS algorithm, and for the exact version of Gibbs smoothing (but not for the one- 
step-late version). The basic tool is the analytical proof of Miilthei and Schorr [75], 
which is actually quite versatile, as demonstrated in Sect. 9. 


Monotonicity of the Shepp-Vardi EM Algorithm 
Here, the two monotonicity properties of the EM algorithm are exhibited, following 


the proof of Miilthei and Schorr [75]. The first monotonicity property (182) follows 
from the derivation of the E-step of the EM algorithm. However, here a purely 
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analytical proof is explained. Vardi et al. [96] prove the two monotonicity properties 


using the geometric results of Csiszar and Tusnady [23]. 
It is useful to define the operator R on nonnegative vectors x € R‘ by 


[Rx]p = xp [A’ (b/Ax)] | ob ee (185) 


#3 

where b/Ax denotes the vector of componentwise quotients. 

Lemma 1. For all nonnegative x and y, with y strictly positive, 
L(x) < L(y) + KL(R y, x) — KL(R y, y). 


Proof. Note that for all nonnegative vectors x and y, 


m hel £ 
L(x) - LQ”) = — DB; log 7H +0 Xp — yp. 
J 


p=1 


j=l 


Now, for strictly positive y, one may write 


[Ax]; _ ‘ a(j, P) Yp oP 
[Ay]; [Ay]; Yp 


p=1 


For each j, this is a convex combination of the points x,/y,, p = 1,2,...,€. Since 
t +> —log ¢t is convex, then by Jensen’s inequality 


m £ ‘ £ 
a(j, p)y x 
L(x) -LQy) < — 0B; r Pog ae 'y) 
=e = ee iy 


£ 
x 
S —Yp [A’r], log + + Xp — Yp» 
p=! YP 


where in the last step the order of summation was interchanged. The lemma 
follows. a 


Proof of the first monotonicity property (182). In the inequality of the lemma 
above, take y = xl andx = Ry = Rx] = xt" Then, Let!) — L(x) < 
—KL(x kt), ll), a 


Proof of the second monotonicity property (183). Start with 
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wet 
kl) _ [k+1]) — 
KL(x*, x!) — KL(x*, x j= se log —_— ay 
p=l 
£ 
b 
* T 
= S x, log E 7a | 
p=l 


Now, if x* solves (179), then it must satisfy the necessary and sufficient conditions 
for a minimum 


x* 20, VLG")20, s3/VL@")|,=0 forall. p. 


The last condition says that x* (—[A’r*], + 1) = 0, where r; = bj /{Ax*]; for 
all 7, so that if x5 > 0, then [Al r* |, = 1. So, for x5 > 0, write 


rf) \] _ adip)by [Axl 
[4 {ar} |, =X [Ax], [Axl 


=1 


which is a convex combination of the points [Ax*]; /[Ax!]; , so by the concavity 
of the logarithm, 


£ m 
b “ p)b; [Ax"]j 
* T J 
dp log E Axil | >) 0% Ax*]; log [Axl]; 

p=1 p=1 j=l 

> 3 b; log i = KL(b, Ax!) — KL(b, Ax kT), 
= 7 [Axel]; 
; k] _ [K+] 
where the last equality follows from )) xp = }0 xp | = YO Dj. a 


Monotonicity for Mixtures 


Here, the two monotonicity properties of the EM algorithm for mixtures of known 
densities are discussed. The difference with the Shepp—Vardi EM algorithm is that 
the system matrix is not normalized to have unit column sums. It will transpire that 
this does not make any difference. 

Recall that the problem is to estimate the pdf 


m 


fyO) =o xj aj), y ERS, (186) 


j=l 
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where the a; are known pdfs and x, is an unknown probability vector, given a 
random sample Y;, Y2,..., Y, of the random variable Y with density fy. Define the 
matrix A € R"*”" by 


Ajj = aij = a;(¥;) foralli and j. (187) 
The EM algorithm for estimating x, is, starting from the uniform vector x!, 
xUTH = tl, (188) 


where the iteration operator M is defined as 


1 n dj; 
Mx\|; =x;- P= 1,.2,..45™M. 189 
[ x]j Xj - a J m (189) 


One begins again with deriving the majorizing function inequality. However, first 
replace the maximum likelihood problem (44) by the equivalent 


; 1 n m m 
minimize L,(x) = — a log x xy ay | + 2 xj 
i=l j=l j= (190) 


subjectto x > 0. 
Note that the constraint that x be a probability vector was traded for the added sum 
in the objective function. 
The majorizing function inequality is the same as before, as is its proof. Then, 


the first monotonicity property follows. 


Lemma 2. /f x and y are nonnegative probability vectors, with y strictly positive, 
then 


Ln(x) < Ln(y) + KL(M y, x) — KL(M y, y). 
Note that the minimizer of the right-hand side (over x) is x = My. 


Lemma 3. Starting from a strictly positive x], the iterates of the EM algo- 
rithm (188) satisfy 


Lie) = 1,.0*™") = EL@ET, x) = 0, 


The second monotonicity property is the same also, but there is a slight change 
in its proof. 
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Lemma 4. Let x* be a solution of (190). Starting from a strictly positive x", the 
iterates of the EM algorithm (188) satisfy 


KL(x*, x!) = KL(x*, xt) > Ly(x) = L,(s*) >0. 
Proof. Since < =). il = I, one has as usual 


[k+1] 

[ky [kKAI}) Xp 

KL(x*, x!) — KL(x*, x j= ae log —_ 7a 
j= 


= Jo xf lo s[4"} Serf | 


Now, if x* solves (179), then it must satisfy the necessary and sufficient conditions 
for a minimum 


x" 20, VLa(Qx*) 20, x7 [VLj(x*)]; =0 forall 7. 


The last condition says that x7 (—[ATr*]; +1) = 0, where r* = (1/n)/[Ax*]; 
for all i, so that if x > 0, then [ATr*]; = 1. So, for x7 > 0, write 


(1/n) “. (1/n)aj;  [Ax*], 
lA" | Axil | = yy a . [Axl], ’ 


which is a convex combination of the points [Ax*]; /[Ax"!]; , so by the concavity 
of the logarithm, 


ae (1/n) (1/n) aij [Ax*]j 
Daf tos| 4” | Soarf > oat Ge ay 


j=l i=l 
[Ax*]; [k] x [k] x 
> Dun log ———— [Axel +s — x7 = Ly (x) — Lax"), 
: [k] _ * 
where the last equality follows from )/ x7) = )) x7 = 1. a 


The convergence of the iterates of the EM algorithm follows. 


Monotonicity of the Smoothed EM Algorithm 


Here, the monotonicity properties of the NEMS algorithm for the smoothed maxi- 
mum likelihood problem (158) are proved. The problem is 
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def 1 a 
minimize L,(f) = = > log{K N(f)|(W) +f f(y) du(y) for 
i=1 


subjectto f € L'(R%), f 20. 


Since this is an infinite-dimensional problem, showing that the iterates of the NEMS 
algorithm converge to a solution of (191) is a bit more involved. In particular, it 
requires us to show the existence of solutions. The only remarkable thing about 
the proofs of the two monotonicity properties for the NEMS algorithm is that apart 
from a few cosmetic changes, they are exactly the same as for the Shepp—Vardi EM 
algorithm. The argument follows Eggermont [34]. 

The NEMS algorithm (159) may be represented as 


Seti = Tht,  fe+1 = Sh Skt, (192) 


starting from a strictly positive initial guess f,, assumed to be a pdf. Here, the map 
7T;, is defined as 


n 


MAI = WNC - . y , k(W; - 2) 
i=1 


KN (AW) 


(193) 


The claim is now that the iterates of the NEMS algorithm satisfy the same two 
monotonicity properties (182) and (183). The crux is again an analytical proof of 
what amounts to the E-step of the EM algorithm. 


Lemma 5. For all densities g and w, 


Ln(g) < Ln) + KL(Sa Tay, g) — KL(Sn Thy, VY). 


Proof. Similar to the proof of Lemma |, one gets that 


WI) 


wwe “@: 


Ln(y) ~ Ln) <= fT) toe 
Since 


log (IV (g)]@)/IN (W)1@)) = [Si log (9/W)] ©), 


and S;, is a symmetric operator, then 


W@l® 
WW)](2) 


dui) =~ f 1S THNG) tos SE% du. 


— [Tie og 70 
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and the lemma follows. || 


Lemma 6. The iterates fi, generated by the NEMS algorithm (192) satisfy 


Ln( fk) — Ln( fei) 2 KLCfe+1, Sc) 2 9. 
Proof. InLemma 5, take g = fx41 and w = fe. Then, SpTnw = f+. a 
The second monotonicity property is actually a little bit stronger than the one for 
the Shepp—Vardi EM algorithm. Note that by the convexity of the KL function jointly 


in both its arguments, KL(S;,g, Spw) < KL(g, Wy). 


Lemma 7. Let f* bea solution of (191), with KL(f*, fi) < oo. Then, the iterates 
Jk generated by the NEMS algorithm (192) satisfy 


KL(f*, fi) — KLCA™, fet) 2 KLCP™, fc.) — KL f*, Th fc) 
= Li( fr) => Li(f*) = 0. 


Proof. Start in the usual fashion and obtain 


Lin( fe) — Ln(f*) = : y log EARS 


i=1 


1 SRO) | IRA SON) 
> IKN FW) 8 TN FM) 


I KO =2 INEM 
[, WI ENF I@ 2 EN Amy 1: 


= 


Now, one would like to get a convex combination, so multiply and divide by the 
sum of the weights k(W; — z)/[K.N f*](W;). Then, the concavity of the logarithm 
gives that the last expression is dominated by 


l n k(W; — 2) 
wx le KW -2 n — TKN fil(2) 
[ WI18-> Do aearmey 8| aut. 


i=1 


k(W; —z) 
n d [KN f *](z) 


i=l 


Now, this expression may be cleaned up as 


; TAO WC 
[to ve Re taro) HO 
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After splitting up the logarithm, note that 


* [Th fx] (2) aes * 
[Fe 108 (ARS) aus) = -KLUT I" Tf 


because 7; is a pdf if gy is one and also 


re ON peel eae a 
[tire we (FATS) ane =f trier, 08 ]io duce 


_ " f*(y) 
= [iT F710) tos 


du(y) = KL(f™, fi), 
since S;,J;, f* = f*. Putting all of this together shows that 


Li( fk) = L(y") < KL(f*, tk) = KL(Tn f*. Th Sk): 
and the lemma follows. || 


The two monotonicity properties imply that the NEMS algorithm converges to a 
solution of the smoothed maximum likelihood problem (191) and that this problem 
actually has a solution. 


Theorem 2. The smoothed maximum likelihood problem (191) has a solution f* € 
Li (R*). 


Proof. One first shows that L,,(f) is bounded from below. Let f be a pdf on R@ 
such that [CNV f](W;) > 0 for alli. Let 1 € R” be the vector of all ones, and let 
vi = [KN f](W;). Then, 


in(f) = KL (2 1.¥) = 2 3 loa NH) + [fod duO. 
Now, by convexity [Vf ](2) < [S;f (2) for all z, so that 
INA = fk DWI due 
< [KOH - 215. flO due 


o / f) / k(W; —2) Si(e— y) duty) due) 
Rd Rd 


ear [ FO) du) = 
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where 


ee / k(W: — 2) Si(z—y) duly) dul) < sup Sh(2), 
R¢ yeR¢d 


yeR¢ 


since k is a pdf. Since S;,(z) = h~“S(h~!2), the boundedness of S then gives that 
Ln < 00 for fixed h > 0. It follows that L,,(f) is bounded from below. 

Now, let {g;,}; be a minimizing sequence for L,,(f). Apply one step of the NEMS 
algorithm to each gx, so We = SaTngx , k = 1,2,... . By the first monotonicity 
property, then {1%}; is a minimizing sequence also. Since each 7;,g, is a pdf, 
then the ~, are uniformly continuous on R?, and so it has a subsequence which 
converges in the strict topology, i.e., uniformly on every compact subset of R4, say, 
with limit y*. This is the Arzela—Ascoli theorem for the strict topology; see [2]. 
Then, along this subsequence 


[KN (Wic)](Wi) — TKN"), 
and it follows that again along this same subsequence 


Ly (Wx) —— Ly (w*). 


Since the whole sequence {1%}; was a minimizing sequence, this shows that y* 
solves the problem (191). | 


Theorem 3. For f''! strictly positive with KL(f*, f'!!) < 00, the NEMS algorithm 
converges to a solution of (191). 


Proof. The proof is just about the same as for the discrete EM algorithm. Thus, 
the first monotonicity property shows that {L,,(f;)} is decreasing. The second 
monotonicity property shows that {KL(f*, f,)}x is decreasing as well and so has 
a nonnegative limit. But then KL(f*, f,) — KL(/*, fc+1) converges to 0, so that 
again the second monotonicity property gives that the NEMS sequence { f;}, is a 
minimizing sequence. All one has to do is extract a convergent subsequence, but that 
follows from the argument in the proof of the existence of convergent subsequences. 
Thus, there exists a subsequence which converges to some element f** in the 
strict topology. Then, [KN'(f,) (Wi) —> [KN(f**)|(W;) for all i, and then 
Li( fe) — Ln(f**) initially only along the subsequence, but since {Ln (fx) }x is 
decreasing, then along the whole sequence. Now, if KL(/**, f1) < oo, then apply 
the second monotonicity property with the solution f**, and then one finds that 
KL(f**, fx) —> 0 along the subsequence, but since {KL( f**, f-)}x is decreasing, 
then along the whole sequence. 

If KL(f**, fi) = ov, then for 0 < ¢ < 1 but arbitrary, apply the second 
monotonicity property with the solution f* = e« f* + (1 — «)f**. Then, 
KL (f*, fi) < 00 and KL (f*, fi) converges, and it is easy to see that 
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lim KL(f.*, fe) =KL(f", f**) =0() for e— 0. 


k->oo 


It follows that KL(f**, f,) —> 0. a 


Monotonicity for Exact Gibbs Smoothing 


The two monotonicity properties also hold for penalized maximum likelihood 
estimation with Gibbs smoothing, at least if the M-step of the EM algorithm is 
performed exactly; see (196) below. This is at least approximately the case in the 
approach of Miller and Roysam [73] but not so for the one-step-late approach of 
Green [49]. 

Here, the monotonicity properties are proved for “arbitrary” Gibbs functionals. 
So, consider the maximum penalized likelihood problem for emission tomography 


m £ 
minimize A(x) = -) b; log[Ax]; + > Xp + G(x) 
j=l p= (194) 


subjectto x = 0, 
where G(x) is convex and differentiable and satisfies 


lim G(x) = +00. (195) 


Il x | 00 


Assume that 6 is strictly positive and that A satisfies the usual conditions (126). 
Note that typically, the roughness prior will be of the form AG(x) for some small 
positive parameter A. In the present context, one may as well take A = 1. 

The goal is again to derive the two monotonicity properties. The majorizing 
functional inequality is just about the same as for the Shepp—Vardi EM algorithm; 
see (180). Recall the definition of the operator R from (185). 


Lemma 8. For all probability vectors x and y, 
A(x) < A(y) + KL(Ry, x) — KL(Ry, y) + G(x) — G(y), 
where r; = b;/[Ay]; for j =1,2,...,m. 


Now, to minimize the right-hand side, set the gradient with respect to x equal to 
0. With y = x*!, this gives the next iterate implicitly as 


k 
a xf far rte, 


*p 1+ [VGGR)], * 


p=1,2,...,£, (196) 
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with zo = b;/[Ax!], for all j. Note that 1 + [VG (aes). > 0 for all p, 
since the equations 


$0 (14[V6 GH], = aB p= Adal 


Dp? 
have a solution (the minimization problem has a solution) and x! is strictly positive 
(by induction). 


The first monotonicity property is almost immediate. The second one takes more 
work. For nonnegative vectors x, y, and w, define 


c 
KL(x, ylw) = >> wp 1 log =? + yp —Xp . (197) 
p=l YP 
Lemma 9. Starting with a strictly positive initial vector x", 
AC) — AGT) > KLOET, ll | lt H) > 0, 
where wkt] = 14+ VG(xk+). 


Proof. From Lemma 8, one gets 


A (x4) — A (x#) 


Al [k+1] 
< > lk +1) xii log —2- ia xi — x! 
p=l ae 
4G (x +H) — G (xl), 
Now use that G(x"*!) — G(x) < (VG(x RTH), AN — lly, fo 


Lemma 10. Let x* be a solution of (194). Then, starting from a strictly positive 
initial guess x!4I, 


KL(v* - x*, wl. xl) — KL * - x*, WRT xd) > Ace) — A(x*) SO, 
where w* = 1+ VG(x*). 
Proof. Write A(x) = L(x) + G(x). So L(x) is the unpenalized negative log- 


likelihood. Now, as in section “Monotonicity of the Shepp—Vardi EM Algorithm,” 
one has 
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ote [Ax*]; + [Axl], lax") | 
J 


: Ax* 
_ wlue les Kg 
Shs E \r toe ar +X, sf 
Pp 
with r= = b;/[Ax*]; for all 7. Now, x* solves the problem (194), so by the 


previous lemma, it must be a fixed point of the algorithm. So, 


* ere 
‘ x5 [A'r*]> 
x° = ——____—__., a eee oo 
TE VGe, 


Then, if x; > 0, one must have [ATr*],/( + [VG(x*)],) = 1. Consequently, by 


convexity 
Ax* Ax* 
r * T * 
treme], ar], 


I+WG@)), ~~ 2 14 1VG@), 


which equals 


yey yer wit kt) whl 
log ( 2 =lo z z + log . 
g xf = e\ oie > a 


[k] 
Wp p Xp P 


Now, substitute this in the upper bound for L(x!) — L(x*). This yields 


£ wht let 
[k]) _ * ok Pp P 
L(x!) — L(x") < y w, x, log oe” ae + 
p= Wp Xp 
» Ww (k] 
* * * 
2. ea et ae 


Now, after some bookkeeping, the first sum is seen to be equal to 


KL (w* «x*, whl. F*) — KL (w* -x*, wht lel) 


£ 
+ fob th ft ap 
p=1 
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Using the inequality log t < t — 1, one gets that 


L(x) — L(x*) < KL(w* - x*, wl. Fl) — KL * - x*, wT. ed) + rem 
(198) 


with the remainder 
£ 
rem = >: {(wE -i) (a - xt) — wy xe + with ane 
p=1 


Now, since wert xett = xf [A’ rl], and likewise for w7 x, then 


£ m £ 
a a [A+] ¥.[A+1] 
wee = op = > wh haa 
j=! p=1 


p=1 


The remaining terms add up to (VG(x!l), x* — xl) which is bounded by 
G(x*) — G(x!) (by convexity). Moving this to the left-hand side of the resulting 
inequality proves the lemma. a 


The convergence of the exact EM algorithm with Gibbs smoothing now follows. 
Lange [62] proves the convergence of the one-step-late version of the algorithm by 
essentially “soft” methods. It would be nice to see under what conditions the two 
monotonicity properties carry over to this version. 


9 EM-Like Algorithms 


The analytical proofs of the inequalities of Lemmas | and 5 may be extended 
to other interesting minimization problems. Rather surprisingly, some of these 
algorithms enjoy the “same” two monotonicity properties as the EM and NEMS 
algorithms (and the proofs appear to be simpler). The problems under consideration 
are “positive” least-squares problems and minimum cross-entropy problems. The 
main idea is that of majorizing functions, as originally exploited by De Pierro [29] 
in the maximum penalized likelihood approach for emission tomography. 

Again, it would have been nice to also outline the geometric approach of Csiszar 
and Tusnady [23], which, just like the analytical approach of Miilthei and Schorr 
[75], is applicable to the minimum cross-entropy problems. However, it is not clear 
that the Csiszar—Tusnady approach works for the “positive” least-squares problem: 
The Kullback—Leibler distance shows up in the monotonicity properties. This is in 
effect due to the multiplicative nature of the algorithms, as explained in the last 
section. 
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Minimum Cross-Entropy Problems 
Consider again the system of equations 
Ax = b, (199) 


in the emission tomography setup (see (126)), with b a nonnegative vector. The 
interest is in the following minimization problem: 


def 


minimize CE(x) = KL(Ax,b) subjectto x eR’, x >0. (200) 
Here, “CE” stands for cross-entropy. (Why it makes sense to consider KL(Ax, b) 
instead of KL(b, Ax) or even || Ax — b ||? is not the issue here.) 

The objective is to obtain a majorizing function for CE(x) that would result in 
a nice algorithm satisfying the “two” monotonicity properties similar to the EM 
algorithm. 


Prejudicing the proceedings somewhat, it is useful to define the operator R on 
nonnegative vectors by 


[Ry]p = Vp exp (Ge =| ) pH 1,2 .c (201) 
Yip 


Here, and elsewhere, A’ log(Ay/b) = A’v, with v; = log([Ay];/b;) for all /. 
Lemma 11. For all nonnegative x, y € R’, 
E(x) < CE(y) + KL(x, Ry) — KLQ, Ry). 
Proof. The starting point is the straightforward identity 
CE(x) = CE(y) + KL(Ax, Ay) + (x—y, ATT). 

with r; = log([Ay];/b;) for all /. 

Now, by convexity of the KL function jointly in both its arguments, the 
conditions (126) and (127) on A imply that 

KL(Ax, Ay) < KL(x, y). (202) 
Finally, since [A7r], = —log ([Ry]p/¥p) , for all p, then 
KL(x, y) + (x—y, ATr) = KL(x, Ry) — KL(y, Ry). 


This completes the proof of the lemma. a 
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The inequality of the lemma immediately suggests an iterative algorithm for the 
minimization of CGE(x). Minimizing the right-hand side gives the optimal x as x = 
Ry with R given by (201). Thus, the iterative algorithm is, starting from a strictly 
positive x!!] € R®, 


gill = [Rx ],, p=1,2,.... (203) 


This is the simultaneous multiplicative algebraic reconstruction technique (SMART 
algorithm). It first appeared in [86]; see (also) Holte et al. [57] and Darroch and 
Ratcliff [25], who called it the iterative rescaling algorithm. The row-action version 
(MART) originated with Gordon et al. [48]. Byrne [7] developed block-iterative 
Versions; see section “The MART and SMART Methods.” The starting point of 
Censor and Segman [18] was entropy maximization subject to the linear constraints 
Ax = b and arrived at various versions of MART including simultaneous and block- 
iterative versions. 
Onto the two monotonicity properties, the first one is immediate. 


Lemma 12. /f x!) is strictly positive, then the iterates of the SMART algo- 
rithm (203) satisfy 


fen) (x!*1) _c (x41) > KL (x, xlet i), 
Note the difference with the first monotonicity property (182) for the Shepp-— 
Vardi EM algorithm. The second monotonicity property is equally simple, but the 


precise form must be guessed. (Actually, it follows from the proof.) 


Lemma 13. /f x* is a solution of the nonnegatively constrained least-squares 
problem (207), then, with x"] strictly positive, 


KL a) _ KL (x*, x41) > Goa) _G& (x*) > 0. 
Proof. Observe that 


KL (x xl) — KL (x*, en) 


Gi let 
= * log 22 [ke] yle-+1] 
“os log aa aa) Mee | 


U [k] U [k] 
= kK] _ y* Xp [k] Xp [k+1] _ ,[k 
= (a) xs) log aT > \s log a er Xp ¢ (204) 
p= Xp p=i Xp 


The last sum equals KL(x"!, x*+"l), and by Lemma 12, then 


KL (xl), xl) > Ce (xl) — a (ll), 
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The first sum equals 


Axtkl 
> (x8 - x3) [ar log ; = (xl x* , Vae(xll)), 
P 


p=1 


where {- , - ) denotes the inner product on R‘ and VQ is the gradient of CE(x). 
By the convexity of CE, then 


(x — x* | VCE (x!) > ce (x!) — GE (x*). 
Summarizing, the above shows that 
KL (x*, x1) — KL (x*, x41) > cB (x!) — CR (x*) + CR (x44) — (x) 
= CE (x**1) — a (x*). 
This is the lemma. a 


As before, the convergence of the SMART algorithm follows starting from any 
strictly positive vector x!!], 


Minimizing Burg’s entropy: Compared with the minimum cross-entropy problem 


from the previous section, the case of Burg’s entropy is problematic. For the positive 
system 


Ax =b 
with the normalization (126), the minimum Burg entropy problem is 


a bj bj 
minimize B E(x) = =log 2 
X [Ax]; [Ax]; (205) 


subject to x = 0. 


The first thing one notices is that it is not a convex problem. Then, it is conceivable 
that the solution set is not convex, and so a “second” monotonicity property is 
not likely to hold. However, there is a majorizing function, which suggests a 
multiplicative algorithm, and there is a “first” monotonicity property. 


Lemma 14. For all nonnegative x and y, 


£ 
B(x) < BEL) + > [IAM Aly — [AT Ip AE (xp — yp) 
D 


p=1 
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ae), oe 
" ([Ay];) 47 Tay]; 


J =1,2,...,m 


The algorithm suggested by the lemma comes about by minimizing the upper 
bound on BE(x) with y = xl, the current guess for the solution. This gives the 
minimizer as x = xt, 


1/2 
At lt] 
[k+l] _ Ik), [ Pp 
Xp =X; TaTqe, (206) 
where 
[k] bj Ik] 1 


re a q; = 
F ([axt],)" / [Axl] 


The “first” monotonicity property reads as follows. 


Lemma 15. Starting with a strictly positive initial guess x"", the iterates generated 
by (206) satisfy 

ay) 
felap 


| 
BE (xl) — BE (x!#1) > > [AT gi] a 
p=1 P 


It follows that the objective function decreases as the iteration proceeds, unless 
one has a fixed point of the iteration. It would seem reasonable to conjecture that 
one then gets convergence of the iterates to a local minimum, but in the absence of 
a second monotonicity property, this is where it ends. 

Some reconstructions from simulated and real data are shown in [15]. The proofs 
of the above two lemmas are shown there as well. 


Nonnegative Least Squares 


The absence of EM algorithms for least-squares problems sooner or later had to be 
addressed. Here, consider positive least-squares problems, and as in section “Min- 
imum Cross-Entropy Problems,” one may as well consider them for the discrete 
emission tomography case. Thus, the interest is in solving the problem 


minimize LS(x) = ||Ax—b||? subjectto x > 0. (207) 


Recall the properties (126) and (127) of the nonnegative matrix A € R” x€ and that 
b is anonnegative vector. It is useful to define the operator T on nonnegative vectors 
by 
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[A7>]p 


T — = [eer oe 2 
[ Vp = Vp [AT Ay], ’ P roms (208) 


The following discussion of the convergence of this algorithm follows De Pierro 
[28] and Eggermont [33]. The first item on the agenda is to prove an analogue of 
Lemma |. 


Lemma 16. For all nonnegative x, y € R°, with y strictly positive, 


£ 
IS(x) <1S(y) + ) > [AD], 


(xp —[Ty]p)? (vp = [Ty]p? 
p=1 


[Ty] ITy]p 
Proof. Observe that 
IS(x) = 1S(y) +2(x—y, A’ (Ay —b)) + || A@—y) IP. 


Let z = x — y. Write Az = A {y'/? (z/y'/?)} (with componentwise vector 
operations) and use Cauchy—Schwarz. Then, 


m vA 
| Az? < >> [Ay] [4 (2/9) = D5  ——* 


j=l p= 
Now, consider || Az |]? + 2(z, A’ (Ay — b)). Completing the square gives 


£ 


; ; [av are [AT (4y — bln \" 
|| Az ||’ +2(z, A’ (Ay —b)) a (+>, | 


> Tar ap [A" (Ay — b)J;,. (209) 


For the first sum, note that the expression inside the parentheses equals x, — [Ty]p. 
Also, [A’ Ay],/yp = [A‘b],/[Ty]p, so that takes care of the first sum. For the 
second sum, note that 


2 
[A’ Ay] [A"}] 
2? = [A (Ay — b)R = —— [(y, -», — 
[A’ Ay], Yp [A’ Ay], 
and the expression for the second sum follows. a 


It is now clear how one may construct an algorithm. Take y = x!], a strictly 
positive vector, and minimize the upper bound on LS(x) given by the lemma. Setting 
the gradient equal to 0 gives xt" = T xl or 
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Alb 
xii = le. [ Ip 


D [AT AXE], PH Ay QoL. (210) 


Note that then all x! are strictly positive because A is nonnegative and has unit 
column sums. 

This algorithm is due to Daube-Witherspoon and Muehllehner [26] for emission 
tomography with the acronym ISRA. 

Onward to the monotonicity properties of the algorithm, the following lemma is 
immediate. 


Lemma 17. /f x") is strictly positive, then the iterates of the ISRA algorithm (210) 
satisfy 


2 
xf — xfer) 


£ 
LS (x!) — 1s (xt) = S“ [ATS], ( aT 
p= Xp 


The “second” monotonicity property is a bit more involved than for the EM 
algorithm but still involves Kullback—Leibler distances. Let 


KIS(x, y) = KL(c-x,c-y) + IS(y) —1S(y*), (211) 


where x = y* is any minimizer of || Ax — b ||? over x > 0. Here, c = A’D and the 
dot means componentwise multiplication. 


Lemma 18. /f x* is a solution of the nonnegatively constrained least-squares 
problem (207), then 


1 1 
KiS(c-x*,c- xl) — Kis(c-x*,c-xF th) > 5 Isl) a LS(x*) = 0. 
Proof. As before, one has 


KL(c-x*, c-xl) = KL(e-x*,c-x 1) 


f let 
= >, Cp X, log an + Cp (x# — xt) 
p= Xp 


m [k] 
Xx 
> Do (:- <r) + ep (xp) — xh") 
p 


Cp (xf - a) (xb = xp) 


xi 


(212) 
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Here, in the second line, the inequality log t = —log(t~') => 1— t7~! was used. 
Now, let C € R‘™* be the diagonal matrix with diagonal components 


[A"]> 


Ch.2 = > 
P»P xr 


5 ae PS 
Then, the least expression equals 


Sige — x* ; C (x! _ xr) = 
(acl Ol (lll = lett) y 4 (lll — eC (ell — tH), 
Since C (x! — x+N) = AT(Ax! — b), then 
(xl —x*, Cl —xFt)) = (xl —x*, AT (Axil —b)) 
> 15(xl) — L1s(x*) 
— 2 2 5 


the last inequality by convexity. Finally, 


£ 
(xk+H — xl o¢ (x = xt) ) oe » [A’b], (xf! — xlk+4] y 
: Pp Pp 


p=1 


which by Lemma 17 dominates LS(x!*+!l) — LS(x!}), This shows that 
KL(c- x*, c+ xl) — KL(c-x*,c-x* th) = 5 LS(x1) — 5 LS(x*) 


= LS(x*l) A LS (x44), 
and the lemma follows. || 


The convergence of the algorithm now follows similar to the EM case. 


Multiplicative Iterative Algorithms 


This final section concerns the observation that multiplicative iterative algorithms 
may be constructed by way of proximal point algorithms as in [33] and that 
Kullback—Leibler distances naturally appear in this context. For arbitrary convex 
functions F on R“, one may solve the problem with nonnegativity constraints 


minimize F(x) subjectto x 20 (213) 
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by computing a sequence {x"I},, with x+4I the solution of 
minimize F(x) + (@,) | KL(x1, x) subjectto x 20, (214) 


starting from some x!!! with strictly positive components. Here, wz > 0. One verifies 
that x*+1 satisfies 


1 
[A+1] _ —_ 

x = T+ oa, [VFORM, ” PH 1 Quist (215) 
This is an implicit equation for x*+"), but explicit versions suggest themselves. Note 
that the objective function in (214) is strictly convex, so that the solution is unique, 
assuming solutions exist. Of course, other proximal functions suggest themselves, 
such as KL(x, x!*l). See, e.g., [19]. The classical one is || x — x! ||?, the squared 
Euclidean distance, due to Rockafellar. See, e.g., [13]. 

It is interesting to note that the implicit algorithm (215) satisfies the two 
monotonicity properties. The first one is obvious, 


F(x!) — FOEt) > (@,)7! KLE), E+), (216) 
since x!*+1] is the minimizer of (214). 


For the second monotonicity property, assume that x* is a solution of (213). Note 
that 


e [k+1] 
KL(x*, xl) — KL(x*, 4M) = 7 xp log PE + xh) — xh 
p=1 Xp 


1 
1+, (VF), 


e 
= > x* log 
p=l 


+4 O, kr [VF (x!) 


£ 
> -x* oo, (VEE), +o, xB NV E(t, 
p=1 


= o,(x¥tl — x*, VEE) > @, (Ft) — FO*)). 
To summarize, 
KL(x*, x1) — KL(x*, xF +1) = @, (FET) — F(x*)). (217) 


The convergence of the algorithm follows. Practically speaking, one has to devise 
explicit versions of the algorithm and see how they behave. 
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10 Accelerating the EM Algorithm 
The Ordered Subset EM Algorithm 


It is well known that EM algorithms converge very slowly, even if one wants to 
stop the iteration “early.” For the general EM algorithm of section “The Bare-Bones 
EM Algorithm Fleshed Out,” some attempts at acceleration have been made along 
the lines of coordinate descent methods or more generally descent along groups of 
coordinates. In particular, the M-step (34) is replaced by a sequence of M-steps, for 
J =1,2,...,m, with x; = xl 


minimize &(x|x;-1) = -f ~z (z|x;-1) log fz (z|x) du(z) om 


subjectto x € 4j;, 
and then xk+!] = x,,. Here {¥ 7 i is a not necessarily disjoint division of the 
parameter space 1. See [68] and references therein. In the context of emission 
tomography, the more generally accepted route to accelerating the EM algorithm has 
been via the ordered subset approach of Hudson and Larkin [59]. Without putting 
too fine a point to it, this amounts to partitioning the data space rather than the 
parameter space. The acceleration achieved by these methods seems to be twofold. 
The ordered subset approach allows for more efficient computer implementations 
and, the convergence itself is speeded up. See, e.g., [60]. 

The ordered subset EM algorithm (OSEM) of Hudson and Larkin [59] deals 
with the maximum likelihood problem of emission tomography (130). The starting 
point is to divide the data into blocks, characterized by the sets of indices 
Q(1), Q(2),..., Q(s) such that 


Q(1) U Q(2) U... U Q(s) = {1,2,...,m}. (219) 


However, the sets need not be disjoint. Define the partial negative Kullback—Leibler 
functionals 


L@O= »¥; 1; toe a + Lan]; - ah r=1,2,...,5. (220) 


jEQ(r) [Ax]; 
Note that for all r, 
£ 
2 [Ax]; = pe with a) = > a(j, p). (221) 
fEQ(r) p=1 JEQ(T) 


The OSEM algorithm now consists of successively applying one step of the Shepp— 
Vardi EM algorithm to each of the problems 
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minimize L,(x) subjectto x20. (222) 


To spell out the OSEM iteration exactly, it is useful to introduce the data vectors B, 
and the matrices A, by 


B, = (bj: j €Q(r)) and A,x = ([Ax]; : j € Q(r)). (223) 
Then, L,(x) = KL(B,, A,x), and the OSEM algorithm takes the form 


SE Sa ea Ae lee. BE 1 Qeyt (224) 


where r = k mods (in the range 1 < r < s) and ol! — Bry {Arx™ |, for all g. 
The slight complication of the @,, arises because the matrices A, do not have unit 
column sums. This is fixed by defining the matrices A, by 


[A, ] a [A,]y,) forall gq and p. (225) 


qp 
Now, define L,(y) = L-(x) = KL(B,,A;y), where yy = Oy-gXq for all g. A 
convenient short-hand notation for this is y = a, -x. Now, if x minimizes L,(x), 
then y = a,-x minimizes L,(y) and vice versa. Since the matrices A, have unit 
column sums, the EM algorithm for minimizing L,(y) is 


a ee. Pate ack, 26) 


with al = Brq / [A, yl], for all g. Transforming back gives (224). 


Regarding the convergence of the OSEM algorithm, the best one can hope for is 
cyclic convergence, i.e., each of the subsequences {xl om sli r>1 converges. Proving 
this would be a daunting task. However, as observed by Byrne [9], it is useful to 
consider what happens if the system of equations Ax = b is consistent in the sense 
that 


dx*>0: Ax* =), (227) 


when one should expect convergence of the whole sequence to a nonnegative 
solution of Ax = b. Hudson and Larkin [59] prove that this is so under the so- 
called subset-balancing condition 


Op =A yp, P=l,2,...,€ and r=1,2,...,5. (228) 


That is, the column sums are the same for all blocks. This is a strong condition, 
even if one allows for overlapping blocks of data. Haltmeier et al. [51] make the 
same assumption in the continuous setting. Byrne [11] observed that the condition 
may be relaxed to that of subset-separability: There exist coefficients 8, and y, such 
that 
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Arp = BrYp, F=1,2,...,8 and p=1,2,...,£. (229) 


The convergence proof of the OSEM algorithm (224) under the subset-separability 
condition (229) relies on the two monotonicity properties of the EM algorithm (226); 
see section “Monotonicity of the Shepp—Vardi EM Algorithm.” After translation, 
one gets the following monotonicity properties for (224). 


Lemma 19. Let x* be a nonnegative solution of Ax = b. Starting from a strictly 
positive xUl © VY, then, with r = k mods, 


L(x) — L(x TY) > KL, - x! a, xl) > 0 and 


KL(a, -x*, a, - xl) — KL(a, + x*, 0, -xF4) > L(x"! — L,(x*) = 0. 


Now, if the a, change with r, then one cannot conclude much from the lemma. 
However, under the subset-separability condition (229), one obtains for all nonneg- 
ative x and y 


KL(q, «x, 07+ y) = By KL(y +x, 7+ y), 
and the inequalities of the lemma translate as follows. 


Corollary 1. Under the conditions of Lemma 19 and the subset-separability 
condition (229), 


pe {L, (x) - Letty} > KL(y-xkt" y- xl) > 0° and 
KL(y -x*, yx!) — KL(y-x*, y- x4") > Br! {L, (x) — L,(x*)} 2 0. 


As in Theorem 1, one may conclude that the sequence {y - x!I}, converges to a 
nonnegative solution x** of Ax = b. Note that there is another way of looking at 
this; see Remark 5 at the end of this chapter. 

As remarked, the subset-separability condition (228) is very strong. It fails 
dramatically in the extreme case of the OSEM algorithm when each block consists 
of a single row. In that case, the OSEM algorithm (224) reduces to 


b; 
[k+l] — Vk], Ps _ 
Xp =X, [axl p= 1,2,...,4, (230) 
where j = kmodm. So, x*+!] is a multiple of x"!, and the OSEM algorithm 


produces only multiples of the initial guess x], So, certainly in this case, the 
algorithm does not converge, but more seriously, it does not do anything useful. 

So, what is one to do? Following Byrne [9] (see also [13]), the next section 
turns to row-action methods (where the blocks consist of a single datum), that 
is, the (additive) algebraic reconstruction technique (ART) and the multiplicative 
version (MART) of Gordon et al. [48], as well as block-iterative variants. (The 
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simultaneous version (SMART) of the multiplicative version was already discussed 
in section “Minimum Cross-Entropy Problems.”) This points into the direction of 
relaxation and scaling. After that, the table is set for block-iterative versions of the 
EM algorithm. 


The ART and Cimmino-Landweber Methods 


It is useful to discuss the situation in regard to the so-called algebraic reconstruction 
technique (ART) of Gordon et al. [48], and the Cimmino—Landweber iteration, a 
reasonable version of the original SIRT method. Herman [55] is the authoritative 
source, but see [95] for a comparison with conjugate gradient method. The ART and 
Cimmino—Landweber algorithms were designed to solve systems of linear equations 
of the form 


Ax=b (231) 
with b the measured nonnegative data and A € R”** with nonnegative components 
a(j, p) but not necessarily unit column sums. Of course, for inconsistent systems of 
equations, this must be replaced by the least-squares problem 


minimize || Ax—b ||? subjectto x €R”, (232) 


but in fact, the ART method solves the weighted least-squares problem 


m : 2 
a(j,-),x)—b; 
minimize pa | ( U ) 5 Z | subject to x € R”, (233) 
a Tai. J 

A standard method for the solution of (232) is the Cimmino—Landweber iteration 


xT — yl 4 @, AT(b — Axl), (234) 


for suitably small but not too small positive relaxation parameters @,. It is 
mentioned here for its analogy with the EM algorithm. The Cimmino—Landweber 
iteration is a well-studied method for the regularization of the least-squares prob- 
lem (232) and is itself subject to acceleration; see, e.g., Hanke [52] and references 
therein. 

At the other end of the spectrum is Kaczmarz’ method, which consists of 
sequential orthogonal projections onto the hyperplanes 


H; = {x ER‘ |(alj, -), x)= 5;}. 


Formally, this is achieved by computing the new iterate x"! from the previous 
one x"! by solving 
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minimize || x —x"! |? subject to (a(j, -), x) = bj. (235) 
The iteration then takes the form, with 7 = k modm, 


bj — (aU, +), xB) 
laG. IP 


xr — yl + oy a(j, -) (236) 


for w, = 1. The relaxation parameter w; is included to see whether choices other 
than w;, = 1 might be advantageous. Geometrically, a requirement is 0 < a < 2. 
The choice w, = 0 would not do anything; the choice @, = 2 implements reflection 
with respect to the hyperplane H;. The algorithm (236) with relaxation originated 
with Gordon et al. [48]. 

Typically, one takes the hyperplanes in cyclic order 7 = k mod m, but Herman 
and Meyer [56] show experimentally that carefully reordering the hyperplanes has a 
big effect on the quality of the reconstruction when the number of iterations is fixed 
before hand. The choice of w(= w, for all k) also matters greatly, but the optimal 
one seems to depend on everything (the experimental setup leading to the matrix A, 
the noise level, etc.), so that the optimal w can be very close to 0 or close to 2 or in 
between. 

Byrne [8,9] observes that the scaling of the ART algorithm is just about optimal, 
as follows. Actually, it is difficult to say much for inconsistent systems, other than 
experimentally (see [56]), but for consistent systems, one has the following two 
monotonicity properties, which are reminiscent of the monotonicity properties for 
the Shepp—Vardi EM algorithm. However, they are much less impressive since they 
only hold for consistent systems. Define 


(a(j,-).x)- 5; 
ll acj, -) IP 


Lemma 20. [f x* satisfies Ax* = b, then 


1S ;(x) = j =1,2,...,m. (237) 


LS; (x1) — LS; (xF 41) = @, (2 — @,) LS; (x), 


[| x) — x* |? = xO — x* |? = @, (2 — @,) LS; (x). 


The proofs involve only (exact) quadratic Taylor expansions and are omitted. The 
conclusion is that ART converges in the consistent case if m, = w is constant and 
0 < @ < 2. Following Byrne [8], one notes that the second monotonicity property 
suggests that w,(2—q@,) should be as large as possible. This is achieved by a, = 1. 
In other words, the original Kaczmarz procedure (236) with @, = 1 is optimally 
scaled. However, as already remarked above, a choice other than w, = | may speed 
things up initially. 

Despite the good news that ART is much faster than the Cimmino—Landweber 
type methods, it is still “slow.” Now, in transmission tomography as in emission 
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tomography, the system of equations Ax = b naturally decomposes into a number 
of blocks 


A,x = B,, r=1,2,...,8, (238) 
(see (223)), and then one has the block version of (235) 


minimize || x —x"1||?_ subjectto A,x = B,, (239) 


with the solution 
xletll — yl 4 wy, AT (4,47)! (B, — A, x"), (240) 


where + denotes the Moore—Penrose inverse. Now, computing (47 4,)' w (for any 
vector w) would be expensive, but it seems reasonable that ATA, should be close to 
diagonal, in which case one may just replace it with its diagonal. This leads to the 
algorithm 


xt] — yl 4 o, AT D-'(B, — A, xl), (241) 


where D, is a diagonal matrix with [D,],, = [Aare ‘ 

Now, it turns out that computing A,x is not much more expensive than computing 
asingle (a(j, - ), x) and that the matrices ATA, are very close to diagonal, so that 
one step of the block method (241) practically achieves as much as the combined 
ART steps for all the equations in one block. So, methods that process naturally 
ordered blocks are appreciably faster than the two extreme methods. See [35]. 

It is not clear how to choose the optimal relaxation parameters. Regarding (241), 
it is known that the algorithm converges cyclically provided the blocks and the 


relaxation parameters are chosen cyclically, i.e., if r = k mods and a, = a@,, 
and 
max || J —,A,D;'A4? |, <1, (242) 
l<rss 
then {yikes converges for each r = 1,2,...,5; see [35]. Moreover, if the 


relaxation parameter is kept fixed, say, 
o, =o forall k, (243) 
and denoting the iterates by x!'+/1(@) to show the dependence on w, then 


lim lim x! t*I(qm) = x*, (244) 
a0 k>0oo 
the minimum norm solution of (233), provided the initial guess belongs to the range 
of A’. See [16]. At about the same time, Trummer [94] showed for the relaxed ART 
method (236) that 


EM Algorithms 377 


lim x(a.) = x*, (245) 
k—>oo 
provided 
Co [o) 
wo. > 0, ~ wo; <oo and 2 We = +00. (246) 
k=1 k=1 


Note the difference between (245) and (246). 


The MART and SMART Methods 


Consider again the system of equations Ax = DB as it arises in the PET setting, 
with A and b having nonnegative components and A having unit column sums. In 
section “Minimum Cross-Entropy Problems,” the SMART algorithm was discussed 
for the solution of 


minimize KL(Ax,b) subjectto x > 0, (247) 
Le., 
b 
[k+1] _ \[k4+1], T _ 
i =a, exp ( log zal }: p= 1,234..,4. (248) 


The multiplicative ART algorithm (MART) of Gordon et al. [48] formally arises as 
the multiplicative version of the additive ART algorithm, to wit 


i a(j,p) 

(kK+1] — 5 [k] J _ 

x = xl. | ——___>~—___ , p=l,2,...,€ 
‘ : (aay) 


or equivalently for p = 1,2,...,€ 


b; 
[k+1] _ Ik), il j 249 
Xp Xp * Xp (» a(j, p) log (a, Ly] (249) 


with w, = 1. Again, the relaxation parameter w, was included to explore whether 
choices other than @, = 1 would be advantageous. Byrne [9] observes that 
the MART algorithm typically does not enjoy the same speedup compared to the 
simultaneous SMART version that ART has over Cimmino—Landweber. To get some 
insight into this, it is useful to consider a projection method analogous to the 
Kaczmarz method of orthogonal projections onto hyperplanes. The method in 
question is well known (see, e.g., [17]), 


minimize KL(x,x"!) subject to (aj, -), xl ) = bj. (250) 
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One may approximately solve this as follows. With the unrestricted Lagrange 
multiplier A, one gets the equations log (ae/xt') + Aa(j, p) = 0 for all p, so 
that 


Xp en exp(Aa(j,p)), p=1,2,...,¢. 


To enforce the constraint, take inner products with a(j, -). This results in 


£ 
bj =(a(j,-),.x) = >> a, p) xt! exp(Qa(j, p)), (251) 


p=1 


and one would like to solve this for A. That does not appear manageable, but it can 
be done approximately as follows. Since the a(j, p) and xl are nonnegative, by 
the mean value theorem, there exists a 6, with 


0< 6 <max{a(j,p):1<p <2}, (252) 
such that the right-hand side of (251) equals exp(A 0) (a( ea hs Then, solv- 
ing (251) for A gives the iteration 

xr — yl exp | wa(j, p) log oon ae (253) 
P Dp (a(j, -), xl) 
with w = 1/6. The conservative choice, the one that changes x! the least, is to 
choose @ as small as possible. In view of (252), this gives w = w;, 
1 


max a(j, Pp) 


1<p<t 


o; = (254) 


Note that if A has unit column sums, then one may expect w to be quite large. This 
may explain why the original MART algorithm is not greatly faster than the SMART 
version. In defense of Gordon et al. [48], one should mention that they considered 
matrices with components 0 or 1, in which case w = 1! 

Following Byrne [8], the block-iterative version of (253) is as follows. In the 
partitioned data setup of (223), the BI-MART algorithm is 


Ok B,. 
T r Pp 


for p = 1,2,...,&, where 


a, = max {arp :l<p< t} (256) 
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is the maximal column sum of A,;. One would expect that @, = 1 should be the 
optimal choice. This is the rescaled BI-MART (or RBI-MART) algorithm of Byrne 
[8]. Following the template of section “Minimum Cross-Entropy Problems,” one 
proves the following majorizing inequality and the two monotonicity properties. 
For nonnegative vectors y and w > 0, define R,, y by 


= Be Pie oP 
[Ro¥]p = Vp exp ( " E le rae |) (257) 


for p = 1,2,...,2. 
Lemma 21. For all nonnegative x and y, 


Oy 
KL(4,x, B-) < KL(A;y, B,) + me {KL(x, Roy) — KL(y, Roy)}- 


Note that the minimizer of the right-hand side is x = R,, y. This would give rise 
to the algorithm (255). 


Proof. Recall the identity from section “Minimum Cross-Entropy Problems,” 
KL(A,x, B,) = KL(A,y, B,) + KL(A,x, 4-y) +(x — y, 470), 
with 0; = log ([4, yj / [B,];). Now, a convexity argument gives that 
KL(A,x, Ay) < a, KL(x, y), 
so that one gets the inequality 


KL(A,x, B,) < KL(4,y, B,) + a KL(x, y) +(x — y, ATQ). 


The definition of the operator R,, gives that 


Ro 
gg eg 
Vp a, 


so then, with 0 = 1/a, 


a, KL(x, y) +(x —y, A?) 


=, KL(x, y) — 0(x — y, log Roy ) 


= a, {(1 — @) KL(x, y) + 6 (KL(x, Ruy) — KL(y, Roy))}- 


The last line follows after some lengthy bookkeeping. So, for 9 < 1 or w = 1, the 
conclusion follows. a 
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The first monotonicity property then follows. 


Lemma 22. Forw > 1landr=kmods, 
KL (A,x"l, B,) — KL (A,x**), B,) > = KL (xl, x#+) > 0, 
a) 


The second monotonicity property follows after some work (omitted). 


Lemma 23. /f x* > 0 satisfies Ax* = b, then for allk andr =k mods, 


KL (x*, x!) — KL (x*, x) > © KL (4,x*4, B,) 20. (258) 
a 


Now, regardless of whether @ = 1 maximizes the right-hand side of the 
inequality (258), the presence of the factor a,, which should be small if the original 
matrix A has unit column sums, suggests that the choice mw = 1 in (258) is a 
tremendous improvement over the case w = a@,, which would arise if one ignored 
the non-unit column sums of A,. 


Row-Action and Block-Iterative EM Algorithms 


Attention now turns to the construction of the row-action version of the EM 
algorithm and the associated block-iterative versions. Recall the formulation (130) 
of the maximum likelihood problem for the PET problem as 


minimize KL(b, Ax) subjectto x € Vis (259) 
with b ¢ R” and A € R”** nonnegative, with A having unit column sums. 


Now, construct a row-action version by considering the following iterative pro- 
jection method, where the new iterate x!*+! is obtained by projecting the previous 


iterate x!I onto the hyperplane (a( J+); x) = bj. The particular projection is 
obtained by 
minimize KL(x"!, x) subject to (a(j, -), x) = b;. (260) 


Again, with A, an unrestricted Lagrange multiplier, one must solve the equations 
—x Hl /x, +1+Aa(j, p) = 0, or 


Xp = xl —Lalj, p) xp, PH Vy Qyac asl: (261) 


At this point, one must make the simplification where x, on the right-hand side is 


replaced by xl, This gives the equation 
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xp =x —Aalj,p)x4, p=1,2,...,6. 
To enforce the constraint, multiply by a(/j, p) and sum over p. Then, 


£ 
bj =(a(j,-), x) )—a So aG, py xf] = (1-16) (aGj,-), x"). (262) 


p=1 
where in the last line the mean value theorem was used, for some @ satisfying 
0< 6 <max{a(j,p):1< p<}. (263) 


Solving for A gives the iterative step 


x = (1 —walj, p)) xl +0 xt! ney: (264) 
for p = 1,2,...,¢, where wm = 1/6. In the notation of (223), with some 
imagination, the block-iterative version is then 

xk+il — RK, xl, (265) 
where the operators R,,,- are defined by 

[Rove = (1 -—o 0») Xp +wx, [A; (B,/A,x)], ; (266) 
for p = 1,2,...,&. So now, is considered to be a relaxation parameter. 


The algorithm (266) was obtained by Byrne [8, 9] after carefully examining the 
analogy with MART vs. RBI-SMART. His choice for the relaxation parameter w is to 
take it depending on the block, so w = w, with 


1 
oO, = ———., (267) 
max Grp 
1<p<t 


which he obtained by deriving the two monotonicity properties discussed below. 
Byrne [8, 9] designated the resulting algorithm (266)-(267) as rescaled block- 
iterative EM for maximum likelihood algorithm (RBI-EMML). At about the same 
time, Browne and De Pierro [5] discovered the algorithm (264)-(266). They 
named (264) the RAMLA (row-action maximum likelihood algorithm). For the latest 
on this, see [92]. 

The above considerations strongly suggest that algorithm (266)—(267) is the cor- 
rect one. This is corroborated by practical experience. The following monotonicity 
properties lend even more weight to it. A slight drawback is that they require that 
the system Ax = b has a nonnegative solution. The first item is again a majorizing 
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inequality. Note that the majorizing inequality is suggested by the algorithm, not the 
other way around. Define 


£ 
o L,(x) + )) (1—-@ayy) 1 log <2 + Yp— oy (268) 
p= 


def 


BI-(x,y) = 


Lemma 24. For nonnegative x, y € R°, 
BI, (x, y) < BL-(y, y) + KL(R, y, x) — KL(R, y, y), 
provided w < 1/max{a,,:1< p < £}. 


Proof. Apply Lemma | to w {KL(B,, A,x) — KL(B,, 4; y)}. Also observe that 


ye [A,Xx]q = : Arp Xp, 
q 


and likewise for [A, y],. This gives 
: y 
BI-(x, y) < BL-(y,y) +) [RoY]p log=* + xp — yp, 
Dp 


p=1 


and the lemma follows. Note that the condition on @ is used implicitly to assure that 
Rwy is nonnegative. i 


The first monotonicity property is an easy consequence. 


Lemma 25. For r = k mods and w < 1/ max{a;p : 1 < p < €}, 
L, (x) _ L, (4) > wo! KL (x+1), xl) > 0. 


Proof. Take y = xl and x = Ray = Rx! = xt Then, one gets 
BI (ert, Fl) —BI (xlKI, xl) <- ll 2), so that 


£ 
(Lr (xl) —L, Cll) #) 2 z, (2—@a;p) 1; log = + Vp —Xpp- 
p=1 


Since @ a, < 1, the conclusion follows. || 


The second monotonicity property reads as follows. 
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Lemma 26. Jf x* > 0 satisfies Ax* = b, then with r =k mods, 
KL (x*, xl) — KL (aa) = @ {Ly (x!) — L, (x*)}, 
provided w < 1/max{a;):1< p< £}. 


The lemma suggests that one should take w as large as possible. This is how 
Byrne [9] arrived at the choice (267). 


Proof of Lemma 26. Since x* satisfies Ax* = b, so A,x* = B, for all r, the 
proof is actually simpler than for the original proof for the EM algorithm; see 
section “Monotonicity of the Shepp—Vardi EM Algorithm.” By the concavity of the 
logarithm (twice), one obtains 


log <2 = log | (1 —aa +4 a \] 
my ” LA awas | 


B B 

= r T 7 
fan we (05 ee )>o[a loa} amt | 
r Pp f 2) 


£ 
KL (x*, x) — KL (x* gern) >o 2 x, [Az log ( (B, / Ar afl) ] 


p=1 


so that 


£ 
+ Doap ape, 
p=1 


Now, the first sum equals 


S>[Arx*]q log ([Br]q/[Arx""], ) = LB log ([Brlq/[Arx"], ) . 
q 


For the remaining sums, note that 
¢ £ 


> xin = > (1—wa,p) xl + o xii [Ay (B,/A,x")] | 


p=1 p=1 
L 
= lo Y [Atl], +0 By 
p=l q q 


Putting the two together proves the lemma. a 
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Remark 5. To wrap things up, note that Byrne [9] shows the convergence (in the 
consistent case) of a somewhat different version of (266), which, under the subset- 
separability condition (229), reduces to the OSEM algorithm (224), thus proving the 
convergence of OSEM under subset-separability (in the consistent case). See also 
Corollary 1. 
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Abstract 


The EM algorithm is not a single algorithm, but a template for the construction of 
iterative algorithms. While it is always presented in stochastic language, relying 
on conditional expectations to obtain a method for estimating parameters in 
Statistics, the essence of the EM algorithm is not stochastic. The conventional 
formulation of the EM algorithm given in many texts and papers on the subject 
is inadequate. A new formulation is given here based on the notion of acceptable 
data. 


1 Introduction 


The “expectation maximization” (EM) algorithm is a general framework for 
maximizing the likelihood function in statistical parameter estimation [1-3]. It is 
always presented in probabilistic terms, involving the maximization of a conditional 
expected value. The EM algorithm is not really a single algorithm, but a framework 
for the design of iterative likelihood maximization methods, or, as the authors of 
[4] put it, a “prescription for constructing an algorithm”; nevertheless, we shall 
continue to refer to the EM algorithm. As we shall demonstrate in Sect.2, the 
essence of the EM algorithm is not stochastic. Our non-stochastic EM (NSEM) is a 
general approach for function maximization that has the stochastic EM methods as 
particular cases. 

Maximizing the likelihood function is a well-studied procedure for estimating 
parameters from observed data. When a maximizer cannot be obtained in closed 
form, iterative maximization algorithms, such as the expectation maximization 
(EM) maximum likelihood algorithms, are needed. The standard formulation 
of the EM algorithms postulates that finding a maximizer of the likelihood is 


EM Algorithms from a Non-stochastic Perspective 391 


complicated because the observed data is somehow incomplete or deficient, and 
the maximization would have been simpler had we observed the complete data. The 
EM algorithm involves repeated calculations involving complete data that has been 
estimated using the current parameter value and conditional expectation. 

The standard formulation is adequate for the most common discrete case, in 
which the random variables involved are governed by finite or infinite probability 
functions, but unsatisfactory in general, particularly in the continuous case, in which 
probability density functions and integrals are needed. 

We adopt the view that the observed data is not necessarily incomplete, but just 
difficult to work with, while different data, which we call the preferred data, leads 
to simpler calculations. To relate the preferred data to the observed data, we assume 
that the preferred data is acceptable, which means that the conditional distribution 
of the preferred data, given the observed data, is independent of the parameter. This 
extension of the EM algorithms contains the usual formulation for the discrete case, 
while removing the difficulties associated with the continuous case. Examples are 
given to illustrate this new approach. 


2 A Non-stochastic Formulation of EM 


The essence of the EM algorithm is not stochastic and leads to a general approach for 
function maximization, which we call the “non-stochastic,’ EM algorithm (NSEM) 
[6]. In addition to being more general, this new approach also simplifies much of 
the development of the EM algorithm itself. 


The Non-stochastic EM Algorithm 

We present now the essential aspects of the EM algorithm without relying on 
statistical concepts. We shall use these results later to establish important facts about 
the statistical EM algorithm. For a broader treatment of the EM algorithm in the 
context of iterative optimization, see [5]. 

The Continuous Case 

The problem is to maximize a nonnegative function f : Z — R, where Z is an 


arbitrary set. We assume that there is z* € Z with f(z*) > f(z), for allz € Z. We 
also assume that there is a nonnegative function b : RY x Z — R such that 


f@= [bo 


Having found zk, we maximize the function 


H(z‘,z) = [oe 2y tog (0.30 (1) 
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to get z*+!, Adopting such an iterative approach presupposes that maximizing 
H(z‘,z) is simpler than maximizing f(z) itself. This is the case with the EM 
algorithm. 

The cross-entropy or Kullback-Leibler distance [7] is a useful tool for analyzing 
the EM algorithm. For positive numbers u and v, the Kullback-Leibler distance from 
utovis 


KL (u,v) = ulog~ +v—u. (2) 
Vv 


We also define KL(0,0) = 0, KL(O,v) = v, and KL(u,0) = +00. The KL 
distance is extended to nonnegative vectors component-wise, so that for nonnegative 
vectors a and b, we have 


J 
KL(a,b) =) KL(a;,bj). (3) 


j=! 


One of the most useful and easily proved facts about the KL distance is contained 
in the following lemma; we simplify the notation by setting b(z) = b(x, z). 


i 


Lemma 1. For nonnegative vectors a and b, with b4. = ei b; > 0, we have 


KL(a,b) = KL(a4,b4) + KL(a, =~). (4) 


This lemma can be extended to obtain the following useful identity. 


Lemma 2. For f(z) and b(x,z) as above, and z and w in Z, with f(w) > 0, we 
have 


KL(b(z), b(w)) = KLf@), fw) + KL), (f(O/FO))b)). ——S) 
Maximizing H(z‘, z) is equivalent to minimizing 
G(e', 2) = KL(b(), b(@) — fo. (6) 
where 


KL(b(z*), b(z)) = / KL(b(x, 2), b(x, 2))dx. (7) 


Therefore, 


—f@) = KL), be) — fA) = KLOC), bE) — FE*, 
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or 
(EM) = f(A) = KL(D(), BET) = KLE), FET). 


Consequently, the sequence { f(z“)} is increasing and bounded above, so that the 
sequence {KL(b(z*), b(z**!))} converges to zero. Without additional restrictions, 
we cannot conclude that { f(z*)} converges to f(z*). 

We get z**! by minimizing G(z*, z). When we minimize G(z, z“*'), we get z 
again. Therefore, we can put the NSEM algorithm into the alternating-minimization 
(AM) framework of Csiszar and Tusnady [12], to be discussed further Sect. 11. 


k+1 


The Discrete Case 

Again, the problem is to maximize a nonnegative function f : Z — R, where Z is 
an arbitrary set. As previously, we assume that there is z* € Z with f(z*) > f(z), 
for all z € Z. We also assume that there is a finite or countably infinite set B and a 
nonnegative function b : B x Z — R such that 


f@ =o bG.2. 


x€B 


Having found al , we maximize the function 


H(e,2) = Y> d(x, 2*) log b(x, z) (8) 


xE€B 


to get &t!, 


We set b(z) = b(x, z) again. Maximizing H(z‘, z) is equivalent to minimizing 


G(c*,z) = KL(b(z*), b(z)) — f(2). (9) 
where 
KL(b(z*), b(z)) = )~ KL(b(x, <*), b(x,2)). (10) 
xEB 


As previously, we find that the sequence { f(z‘)} is increasing, and {KL(b(<‘), 
b(z +1))\ converges to zero. Without additional restrictions, we cannot conclude 
that { f(z“)} converges to f(z*). 


3 The Stochastic EM Algorithm 
The E-Step and M-Step 


In statistical parameter estimation, one typically has an observable random vector 
Y taking values in R™ that is governed by a probability density function (pdf) or 
probability function (pf) of the form fy (y|@), for some value of the parameter vector 
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0 € ©, where @ is the set of all legitimate values of 6. Our observed data consists 
of one realization y of Y; we do not exclude the possibility that the entries of y 
are independently obtained samples of a common real-valued random variable. The 
true vector of parameters is to be estimated by maximizing the likelihood function 
L,(@) = fy(y|@) over all 6 € © to obtain a maximum likelihood estimate, Oy7,. 

To employ the EM algorithmic approach, it is assumed that there is another 
related random vector X, which we shall call the preferred data, such that, had 
we been able to obtain one realization x of X, maximizing the likelihood function 
Lx(9) = fx(x|@) would have been simpler than maximizing the likelihood 
function L,(@) = fy(y|@). Of course, we do not have a realization x of X. The 
basic idea of the EM approach is to estimate x using the current estimate of 6, 
denoted 6*, and to use each estimate x* of x to get the next estimate Ok+1, 

The EM algorithm proceeds in two steps. Having selected the preferred data X, 
and having found 6“, we form the function of 6 given by 


Q(8|0") = Ellog fx(x|4)|y. 0"): (11) 
this is the E-step of the EM algorithm. Then we maximize Q(6|6*) over all 6 to get 
6*+1: this is the M-step of the EM algorithm. In this way, the EM algorithm based 


on X generates a sequence {ok } of parameter vectors. 
For the discrete case of probability functions, we have 


Q(0|6") = S70 fei Orly, 6") log fx (x19), (12) 
and for the continuous case of probability density functions, we have 


(810%) = f fxr Gly. 6") tog, fe (xl de. (13) 
In decreasing order of importance and difficulty, the goals are these: 
1. To have the sequence of parameters {6} converging to 6; 
2. To have the sequence of functions { fy (x|@*)} converging to fy (x|@uz); 
3. To have the sequence of numbers {L,(6*)} converging to Ly (Oz); 
4. To have the sequence of numbers {L,(6")} non-decreasing. 
Our focus here is mainly on the fourth goal, with some discussion of the third goal. 
We do present some examples for which all four goals are attained. Clearly, the first 
goal requires a topology on the set ©. 


Difficulties with the Conventional Formulation 


In [1] we are told that 
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Fxyy Oly, 8) = fx x1)/fr 18). (14) 


This is false; integrating with respect to x gives one on the left side and 1/ fy (y|6) 
on the right side. Perhaps the equation is not meant to hold for all x, but just for 
some x. In fact, if there is a function / such that Y = h(X), then Eq. (14) might 
hold for those x such that h(x) = y. In fact, this is what happens in the discrete 
case of probabilities; in that case we do have 


fr(yl0)= D> fx(x18), (15) 
xEh—!{y} 
where 
ho'{y} = {x|A(x) = y}. 
Consequently, 
_ § fx 10)/ fr (10), ifx © AT yy}: 
Ixy aly, 8) = 0. ifx ¢ Ay}. (16) 


However, this modification of Eq. (14) fails in the continuous case of probability 
density functions, since h~'{y} is often a subset of zero measure. Even if the set 
h—'{y} has positive measure, integrating both sides of Eq. (14) over x € h7!{y} 
tells us that fy (y|@) < 1, which need not hold for probability density functions. 


An Incorrect Proof 

Everyone who works with the EM algorithm will say that the likelihood is non- 
decreasing for the EM algorithm. The proof of this fact usually proceeds as follows; 
we use the notation for the continuous case, but the proof for the discrete case is 


essentially the same. Use Eq. (14) to get 


log fx (x|0) = log fxy (x|y, @) — log fr (v9). (17) 


Then replace the term log fy(x|@) in Eq.(13) with the right side of Eq. (17), 
obtaining 


log fy(y|8) — Q(6|6") = — / far@lyo ee forte Ode: C18) 


Jensen’s Inequality tells us that 


[veo toguoax > [ we) log v(x)dx, (19) 
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for any probability density functions u(x) and v(x). Since fy |y(x|y,@) is a 
probability density function, we have 


/ fav Gly, 0) log few (ly, dx < i; fay Ctly, 0 log fa ly, dx. 
(20) 


We conclude, therefore, that log fy (y|@) — Q(6|6*) attains its minimum value at 
6 = 0*. Then we have 


log fy (y|6**") — log fr(v6) = Q6**'|6*) — O* 0) >0. (21) 
This proof is incorrect; clearly it rests on the validity of Eq. (14), which is generally 


false. For the discrete case, with Y = h(X), this proof is valid, when we use 
Eq. (16), instead of Eq. (14). In all other cases, however, the proof is incorrect. 


Acceptable Data 


We turn now to the question of how to repair the incorrect proof. Equation (14) 
should read 


Sxiy xly, 8) = fry (x, y|8)/fy (1), (22) 
for all x. In order to replace log fy (x|0) in Eq. (13), we write 
Ixy (x, yl) = fey ly, OA 9), (23) 


and 


Sxy (x, yO) = fyyx (vx, 8) fx (x/8), (24) 


so that 
log fx (x|0) = log fry (x|y, @) + log fy (18) — log frx(y|x,8). (25) 
We say that the preferred data is acceptable if 
Frix(y|x, 9) = fyix (yx); (26) 


that is, the dependence of Y on X is unrelated to the value of the parameter 0. This 
definition provides our generalization of the relationship Y = h(X). 

When X is acceptable, we have that log fy(y|9) — O(0|6*) again attains its 
minimum value at 6 = 0*. The assertion that the likelihood is non-decreasing then 
follows, using the same argument as in the previous incorrect proof. 
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4 The Discrete Case 


In the discrete case, we assume that Y is a discrete random vector taking values in a 
finite or countably infinite set A, and governed by probability fy (y|@). We assume, 
in addition, that there is a second discrete random vector X, taking values in a finite 
or countably infinite set B, and a function h : B — A such that Y = h(X). We 
define the set 


h7\{y} = {x € BlA(x) = y}. (27) 
Then we have 
fryl0)= D> fxd). (28) 
xehl{y} 


The conditional probability function for X, given Y = y, is 


Sx (x9) 
fr (v0) 


Sx\y Oly, 8) = (29) 


for x € h~'{y}, and zero, otherwise. The so-called E-step of the EM algorithm is 
then to calculate 


Q(6|0*) = E((log fx(X|0)|y.0) = Yo fry Gly. 6*) log fx (x|9), (30) 


xeEh—!{y} 


and the M-step is to maximize Q(0|0*) as a function of @ to obtain 6**!, 
Using Eq. (29), we can write 


O(6|0)= S> fry(ely. O*)log fry (ly, ) + log fr(y19). G1) 


xeEh—l{y} 
Therefore, 


log fr(y|0) — O06) =— Y> fry (aly, O*) log fxly (ly, 8). 


xeh—!{y} 


Since 


> fev@lye)= Do fav@ly,6) =1, 


xeEh—!{y} xEh—!{y} 


it follows from Jensen’s Inequality that 
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— So fei @ly, 6%) log fey Gly. 8) 


xeh—l{y} 
>- Yo feGly, 6%) log fry ly, 0). 
xeh ty} 


Therefore, log fy (y|9) — Q(0|0*) attains its minimum at 9 = 6*. We have the 
following result. 


Proposition 1. The sequence { fy (y|0*)} is non-decreasing. 


Proof. We have 


log fy (y|0**") — Q(6**"|6") > log fy (y1") — O(6*|6"), 
or 


log fy (y|0**!) — log fy (y|6*) = O(0**"6*) — O(6*|6*) = 0. 


Let ¥;-14,}(%) be the characteristic function of the set h—'{y}, that is, 


1, ifx €ho'{y}; 


0, ifx € h7'{y}. a 


Xn-'fy3(X) = 


With the choices z = 0, f(z) = fy(y|@), and b(z) = fx (x|@)Xn-145}(X), the 
discrete EM algorithm fits into the framework of the non-stochastic EM algorithm. 
Consequently, we see once again that the sequence { fy (y|6“)} is non-decreasing 
and also that the sequence 


{KLOET} = EY (KL Sx lO), fx lO**)} 


xeh—l{y} 


converges to zero. 


5 Missing Data 


We say that there is missing data if the preferred data X has the form X = (Y, W), 
so that Y = h(X) = h(Y,W), where h is the orthogonal projection onto the 
first component. The case of missing data for the discrete case is covered by the 
discussion in Sect. 4, so we consider here the continuous case in which probability 
density functions are involved. 
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Once again, the E-step is to calculate 0(6|6") given by 
O(6|9") = E(log fx(X|9)|y, 8"). (33) 


Since X = (Y, W), we have 
Sx(x|9) = frw(y, wl9). (34) 


Since the set h~!{y} has measure zero, we cannot write 


0(6|6*) = i; fav Ctly, 6) log fa (xl0)dx. 
'{y} 


h—- 


Instead, following [8], we write 
200108) = f fr v-wlO) og frwr(y.wldw/ fr. 5) 


Consequently, maximizing Q(6|6") is equivalent to maximizing 


i AGO WO neh OMew 


With b(6) = b(6, w) = frw(y, wl9) and 
fryl0) = f@)= i Poawioaw= / b(8)dw, 


we find that maximizing Q(6|6") is equivalent to minimizing KL(b(0*), b(@)) — 
f(@). Therefore, the EM algorithm for the case of missing data falls into the 
framework of the non-stochastic EM algorithm. We conclude that the sequence 
{ £(0*)} is non-decreasing and that the sequence { KL(b(6*), b(@*+'))} converges 
to zero. 

Most other instances of the continuous case in which we have Y = A(X) can 
be handled using the missing-data model. For example, suppose that Z, and Z» 
are uniformly distributed on the interval [0, 0], for some positive 0 and that Y = 
Z, + Zz. We may, for example, then take W tobe W = Z; — Z2 and X = (Y, W) 
as the preferred data. We shall discuss these instances further in Sect. 7. 


6 The Continuous Case 


We turn now to the general continuous case. We have a random vector Y taking 
values in R% and governed by the probability density function fy(y|6). The 
objective, once again, is to maximize the likelihood function L,(@) = fy(y|@) 
to obtain the maximum likelihood estimate of 6. 
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Acceptable Preferred Data 


For the continuous case, the vector 6+! is obtained from 6* by maximizing the 
conditional expected value 


Q(810) = E(og fe(X19)|y.%) = ffx (aly. 6%) tog fe(xl@)dx. G6) 
Assuming the acceptability condition and using 


fey (x. yO") = fey Oly. OEY A O10"), 


and 


log fx(x|@) = log fy.y (x, y|9) — log fy|x(y|x), 


we find that maximizing E(log fy (x|0)|y, 0*) is equivalent to minimizing 


H(6",6) = : far (2, 16") log facz (x, yl0)ax. G7) 


With f(@) = fy(y|@), and b(@) = fx y(x, y|@), this problem fits the framework 
of the non-stochastic EM algorithm and is equivalent to minimizing 


G(6*,6) = KL(b(6*), b(@)) — f (0). 


Once again, we may conclude that the likelihood function is non-decreasing and 
that the sequence {KL (b(0*), b(6**!))} converges to zero. 

In the discrete case in which Y = h(X), the conditional probability fy|y (y|x, 9) 
is 6(y — A(x)), as a function of y, for given x, and is the characteristic function of 
the set h~'(y), as a function of x, for given y. Therefore, we can write fy|y (x|), #) 
using Eq. (16). For the continuous case in which Y = h(X), the pdf fyy (y|x, @) is 
again a delta function of y, for given x; the difficulty arises when we need to view 
this as a function of x, for given y. The acceptability property helps us avoid this 
difficulty. 

When X is acceptable, we have 


Fxiy ly, 8) = fryx le) fx 1 8)/Fy (V1), (38) 


whenever fy (y|6) 4 0, and is zero otherwise. Consequently, when X is acceptable, 
we have a kernel model for fy (y|@) in terms of the fy (x|@): 


fr) = 1 fixe) fel 0d: (39) 
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for the continuous case we view this as a corrected version of Eq. (15). In the discrete 
case the integral is replaced by a summation, of course, but when we are speaking 
generally about either case, we shall use the integral sign. 

The acceptability of the missing data W is used in [9], but more for computa- 
tional convenience and to involve the Kullback-Leibler distance in the formulation 
of the EM algorithm. It is not necessary that W be acceptable in order for likelihood 
to be non-decreasing, as we have seen. 


Selecting Preferred Data 


The popular example of multinomial data given below illustrates well the point that 
one can often choose to view the observed data as “incomplete” simply in order to 
introduce “complete” data that makes the calculations simpler, even when there is no 
suggestion, in the original problem, that the observed data is in any way inadequate 
or “incomplete.” It is in order to emphasize this desire for simplification that we 
refer to X as the preferred data, not the complete data. 

In some applications, the preferred data X arises naturally from the problem, 
while in other cases the user must imagine preferred data. This choice in selecting 
the preferred data can be helpful in speeding up the algorithm (see [10]). 

If, instead of maximizing 


J frvvcrly. 68) tog fe (alae. 
at each M-step, we simply select 9+! so that 


i few ly, 6*) log fcr (x, yO") ax 
= / Fay Cely, 0) log fay (x, y19*)dx > 0, 


we say that we are using a generalized EM (GEM) algorithm. It is clear from the 
discussion in the previous subsection that whenever X is acceptable, a GEM also 
guarantees that likelihood is non-decreasing. 


Preferred Data as Missing Data 


As we have seen, when the EM algorithm is applied to the missing-data model, 
the likelihood is non-decreasing, which suggests that, for an arbitrary preferred 
data X, we could imagine X as W, the missing data, and imagine applying the 
EM algorithm to Z = (Y, X). This approach would produce an EM sequence of 
parameter vectors for which likelihood is non-decreasing, but it need not be the 
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same sequence as obtained by applying the EM algorithm to X directly. It is the 
same sequence, provided that X is acceptable. We are not suggesting that applying 
the EM algorithm to Z = (Y, X) would simplify calculations. 

We know that, when the missing-data model is used and the M-step is defined 
as maximizing the function in (35), the likelihood is not decreasing. It would seem 
then that for any choice of preferred data X, we could view this data as missing and 
take as our complete data the pair Z = (Y, X), with X now playing the role of W. 
Maximizing the function in (35) is then equivalent to maximizing 


/ few Gly, 6*) log facr (c, yl) ax: (40) 


to get O**!. Tt then follows that L,(6*t!) > L,(0*). The obvious question is 
whether or not these two functions given in (11) and (40) have the same maximizers. 
For acceptable X we have 


log fxy (x, y|@) = log fx (x|@) + log fy\x (yx), (41) 


so the two functions given in (11) and (40) do have the same maximizers. It follows 
once again that whenever the preferred data is acceptable, we have Loe) = 
L,(0*). Without additional assumptions, however, we cannot conclude that {6*} 
converges to Oy, nor that { fy (y|0*)} converges to fy (y|9uz). 


7 The Continuous Case with Y = h(X) 


In this section we consider the continuous case in which the observed random vector 
Y takes values in R%; the preferred random vector X takes values in R™: the 
random vectors are governed by probability density functions fy (y|9) and fy (x|@), 
respectively; and there is a function h : RY — R™ such that Y = A(X). In most 
cases, M > N andh7'!{y} = {x|h(x) = y} has measure zero in R™. 


An Example 
For example, suppose that Z; and Zz are independent and uniformly distributed 
on the interval [0,0], for some 6 > 0 to be estimated. Let Y = Z, + Z2. With 


Z = (Zi, Z2), andh : R? —> R given by h(z1, 22) = z1 + 22, we have Y = h(Z). 
The pdf for Z is 


1 
FSz(2|@) = fz(z1,2|0) = ga X10.0) 1) X10,6) (22). (42) 


The pdf for Y is 
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fr) gw, if0<y <8; (43) 
PME) BD 386 <y 220. 
It is not the case that 
frvl6) = i falc\d), (44) 
a 'fy} 


since h—'{y} has measure zero in R?. 
The likelihood function is L(6) = fy(y|@), viewed as a function of 6, and is 
given by 


(45) 


Therefore, the maximum likelihood estimate of 0 is @yz = y. 

Instead of using Z as our preferred data, suppose that we define the random 
variable W = Zp, and let X = (Y, W), a missing-data model. We then have Y = 
h(X), where h : R* — R is given by h(x) = h(y,w) = y. The pdf for Y given in 
Eq. (43) can be written as 


1 
fy (y|@) = / 92 X00 (vy — w)x(0,6)(w)dw. (46) 
The joint pdf is 
1/02, f <y<64w; 
fr wy, wl@) = ‘ a = is (47) 
( otherwise. 


Censored Exponential Data 


McLachlan and Krishnan [1] give the following example of a likelihood maximiza- 
tion problem involving probability density functions. This example provides a good 
illustration of the usefulness of the missing-data model. 

Suppose that Z is the time until failure of a component, which we assume is 
governed by the exponential distribution 


fGI8) = ne7, (48) 


where the parameter 0 > 0 is the expected time until failure. We observe a random 
sample of N components and record their failure times, z,. On the basis of this data, 
we must estimate 0, the mean time until failure. 
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It may well happen, however, that during the time allotted for observing the 
components, only r of the N components fail, which, for convenience, are taken 
to be the first 7 items in the record. Rather than wait longer, we record the failure 
times of those that failed and record the elapsed time for the experiment, say 7’, for 
those that had not yet failed. The censored data is then y = (y1,..., yn), where 
Vn = Z is the time until failure forn = 1,...,r,andy, = T forn =r-+1,...,N. 
The censored data is reasonably viewed as incomplete, relative to the complete data 
we would have had, had the trial lasted until all the components had failed. 

Since the probability that a component will survive until time T is e~"/°, the pdf 
for the vector y is 


r 


fr(y|0) = (T] se) eo N—)T/O | (49) 


n=1 


and the log likelihood function for the censored, or incomplete, data is 


N 
1 
log fr (yl®) = —r log@ — 5 ) Yn: (50) 
n=1 


In this particular example, we are fortunate, in that we can maximize fy (y|9) easily, 
and find that the ML solution based on the incomplete, censored data is 


he ees N-r 
Outi = = yn = — Dn + ——T. (51) 
n=1 


n=1 


In most cases in which our data is incomplete, finding the ML estimate from the 
incomplete data is difficult, while finding it for the complete data is relatively easy. 

We say that the missing data are the times until failure of those components that 
did not fail during the observation time. The preferred data is the complete data 


xX = (z,...,Zy) of actual times until failure. The pdf for the preferred data X is 
a 
fx (x18) = I] re (52) 


and the log likelihood function based on the complete data is 


N 
1 
log fx(x|0) = —N log — 5 D1 an. (53) 


n=1 


The ML estimate of 6 from the complete data is easily seen to be 


1 N 
OmLe = N > Zn- (54) 


n=1 
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In this example, both the incomplete-data vector y and the preferred-data vector x 
lie in RY. We have y = h(x) where the function h operates by setting to T any 
component of x that exceeds 7. Clearly, for a given y, the set h~'{y} consists of all 
vectors x with entries x, > T or x, = y, < T. For example, suppose that N = 2, 
and y = (91,7), where y; < T. Then 7! {y} is the one-dimensional ray 


Ao'{y} = {x = (1, X2)| x2 = T}. 


Because this set has measure zero in R?, Eq. (44) does not make sense in this case. 

We need to calculate E(log fy (X|@)|y, 6*). Following McLachlan and Krishnan 
[1], we note that since log fy(x|@) is linear in the unobserved data Z,,n = r+ 
1,...,.N, to calculate E(log fy(X|6)|y, 0), we need only replace the unobserved 
values with their conditional expected values, given y and 6*. The conditional 
distribution of Z, — 7, given that Z, > T, is still exponential, with mean 0. 
Therefore, we replace the unobserved values, that is, all the Z,, forn = r+1,...,N, 
with T + 60". Therefore, at the E-step we have 


N 
E(log fy(X|@)|y, 0“) = —N log 6 — 5((22») +(N - net) (55) 


n=1 


The M-step is to maximize this function of 0, which leads to 


N 
oft) = ( Yim) + - ne) /N. (56) 


n=1 
Let 0* be a fixed point of this iteration. Then we have 
N 
g* = ((2») N= ner) 
n=1 
so that 
ia 
0* = - ae 
n=1 


which, as we have seen, is the likelihood maximizer. 


A More General Approach 


Let X take values in RY and Y = h(X) take values in R”, where M < N and 
h: RX — R™ isa (possibly) many-to-one function. Suppose that there is a second 
function k : RY — RY such that the function 
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G(x) = (h(x), k(x)) = (y,w) = u (57) 


has inverse H(y, w) = x. Denote by J(y, w) the determinant of the Jacobian matrix 
associated with the transformation G. Let 


W(y) = {wlw = k(x), andy = h(x)}. 


Then 


fr(10) = f 


wEeWw(y 


fe HO.mIQ.wdw, (58) 


Then we apply the missing-data model for the EM algorithm, with W = k(X) as 
the missing data. 


8 A Multinomial Example 


In many applications, the entries of the vector y are independent realizations of a 
single real-valued or vector-valued random variable V, as they are, at least initially, 
for finite mixture problems to be considered later. This is not always the case, 
however, as the following example shows. 

A well-known example that was used in [11] and again in [1] to illustrate the 
EM algorithm concerns a multinomial model taken from genetics. Here there are 
four cells, with cell probabilities 5 + 790, 7. — 00), 7. — 0), and 590, for some 
6 € © = (0, 1] to be estimated. The entries of y are the frequencies from a sample 
size of 197. We then have 


197! LP! we _ ee " 
FeV) = alg + 79" Ga OP Ga-OyrGar". 69) 


It is then supposed that the first of the original four cells can be split into two 
subcells, with probabilities 5 and 790. We then write yj = yi + yi, and let 


X = (Yu, Yi2, Yo, Y3, Ya), (60) 


where X has a multinomial distribution with five cells. Note that we do now have 
Y =h(X). 

This example is a popular one in the literature on the EM algorithm (see [11] 
for citations). It is never suggested that the splitting of the first group into two 
subgroups is motivated by the demands of the genetics theory itself. As stated in 
[1], the motivation for the splitting is to allow us to view the two random variables 
Yio + Y4 and Y2 + Y3 as governed by a binomial distribution; that is, we can view 
the value of y12 + y4 as the number of heads and the value y2 + y3 as the number 
of tails that occur in the flipping of a biased coin yj2 + ya + y2 + y3 times. This 
simplifies the calculation of the likelihood maximizer. 
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9 The Example of Finite Mixtures 


We say that a random vector V taking values in R? is a finite mixture if, for j = 
1,...,J, fj is a probability density function or probability function, 0; > 0 is 
a weight, the 6; sum to one, and the probability density function or probability 
function for V is 


J 
fv (vl) = 95; f7(). (61) 
j=l 


The value of D is unimportant, and for simplicity, we shall assume that D = 1. 

We draw N independent samples of V, denoted v,, and let y,, the nth entry of 
the vector y, be the number v,,. To create the preferred data we assume that for each 
n, the number v, is a sample of the random variable V,, whose pdf or pf is f;,, where 
the probability that j, = j is 0;. We then let the N entries of the preferred data X 
be the indices j,. The conditional distribution of Y, given X, is clearly independent 
of the parameter vector 6 and is given by 


N 
fuxQlx, 0) = [] fi, On); 
n=1 


therefore, X is acceptable. Note that we cannot recapture the entries of y from those 
of x, so the model Y = h(X) does not hold here. Note also that although the vector 
y is taken originally to be a vector whose entries are independently drawn samples 
from V, when we create the preferred data X , we change our view of y. Now each 
entry of y is governed by a different distribution, so y is no longer viewed as a 
vector of independent sample values of a single random vector. 


10 The EM and the Kullback-Leibler Distance 

We illustrate the usefulness of acceptability and reformulate the M-step in terms of 
cross-entropy or Kullback-Leibler distance minimization. 

Using Acceptable Data 

The assumption that the data X is acceptable helps simplify the theoretical 
discussion of the EM algorithm. 


For any preferred X the M-step of the EM algorithm, in the continuous case, is 
to maximize the function 


/ fay Cely, 0) log fa (xl), (62) 
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over 6 € ©; the integral is replaced by a sum in the discrete case. For notational 
convenience we let 


b(O*) = fe(rly, *), (63) 


and 


F(8) = fx (x18); (64) 


both functions are functions of the vector variable x. Then the M-step is equivalent 
to minimizing the Kullback-Leibler or cross-entropy distance 


OF 
KL(b(6"), (0) = / fav oly. 6*) og (SEE) 


2 fry (ly, 0) 
= | fav (sly. 08) og (A) 
+ fx (x0) — fey (ly, O)dx. (65) 


This holds since both fy (x|@) and fyjy (x|y, 6") are probability density functions 
or probabilities. 
For acceptable X we have 


log fx.y (x, y|@) = log fxly ly, 9) + log fy (y|) 
= log fy|x (y|x) + log fx (x|6). (66) 


Therefore, 


log fy (y|0**!) — log fy (90) = KL(b(6"), f(0)) — KL(b(0*), f(O**")) 
+ KL(b(6*), b(0**')) — KL(b(6*), b(6)). 


(67) 
Since 6 = 6*+! minimizes KL(b(6*), f(@)), we have that 
log fy (y|6**') — log fr(v16") = KL(b*), f(6*)) 
— KLO(6"), FO") 
+ KL(b(6*), b(@**!)) = 0. (68) 


This tells us, once again, that the sequence of likelihood values {log fy (y|6*)} is 
increasing and that the sequence of its negatives, {— log fy(y|6")}, is decreasing. 
Since we assume that there is a maximizer Oy, of the likelihood, the sequence 
{—log fy (y|6*)} is also bounded below and the sequences {KL(b(0*), b(@**!))} 
and {KL(b(6"), f(0")) — KL(b(0*), f(0**'))} converge to zero. 
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Without some notion of convergence in the parameter space ©, we cannot 
conclude that {9*} converges to a maximum likelihood estimate 9y,,. Without some 
additional assumptions, we cannot even conclude that the functions f(6“) converge 


to f(Omz). 


11 The Approach of Csiszar and Tusnady 


For acceptable X the M-step of the EM algorithm is to minimize the function 
KL(b(6*), f(@)) over 6 € © to get 6**!. To put the EM algorithm into the 
framework of the alternating-minimization approach of Csiszar and Tusnady [12], 
we need to view the M-step in a slightly different way; the problem is that, for the 
continuous case, having found 6*+!, we do not then minimize KL(b(6), f(0*+!)) 
at the next step. 


The Framework of Csiszar and Tusnady 


Following [12], we take Y(p, q) to be a real-valued function of the variables p € P 
and q € Q, where P and Q are arbitrary sets. Minimizing Y(p,q") gives p” and 
minimizing U(p",q) gives q”*!, so that 


W(p",q") > W(p",g"*!) > Wie ge): (69) 


The objective is to find (p, g) such that 


V(p.g) = VA.q), 
for all p and q. In order to show that {W(p”, q”)} converges to 


d= inf pee) 


pEP.qe 


the authors of [12] assume the three- and four-point properties. 
If there is a nonnegative function A : P x P — R such that 


(p,q!) — U(p"*,g"*") = ACp, p"*"), (70) 
then the three-point property holds. If 
A(p, p") + V(p.q) = Up. q"*"), (71) 


for all p and q, then the four-point property holds. Combining these two inequalities, 
we have 


A(p, p") — A(p, p"*') = U(p"t!,g"*!) — (p,q). (72) 
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From the inequality in (72) it follows easily that the sequence {V(p”, g”)} converges 
to d. Suppose this is not the case. Then there are p’, q’, and D > d with 


W(p",q") > D> U(p',q') > d. 
From Eq. (72) we have 
AG 2 Y= AG 2 UR a =9e a 2D=U0' a) > 6. 
But since {A(p’, p”)} is a decreasing sequence of positive quantities, successive 
differences must converge to zero; that is, {¥(p"*!,g"*!)} must converge to 


W(p’, gq’), which is a contradiction. 
The five-point property of [12] is obtained by combining (70) and (71): 


U(p,q) + V(p.q"') = U(p.q") + U(p", 4g"). (73) 
Note that the five-point property does not involve the second function A(p’, p). 
However, assuming that the five-point property holds, it is possible to define 


A(p’, p) so that both the three- and four-point properties hold. Assuming the five- 
point property, we have 


WU(p.q" ')— U(p,q") = V(p",q") — Vp. gq). (74) 


from which we can show easily that {Y(p", q”)} converges to d. 


Alternating Minimization for the EM Algorithm 


Assume that X is acceptable. We define the function F'(@) to be 


F(6) = / fay Gly, 0) log fix Olxdx, (75) 


for the continuous case, with a sum replacing the integral for the discrete case. Using 
the identities 


fy (x, yIO) = fry oly, O Sr (19) 
= fyix(y|x, 8) fy (X10) = fyjx (vx) fx (19), 


we then have 
log fy (y|9) = F(0") + KL(b(6’), b(6)) — KL(b(O’), f()), (76) 


for any parameter values 0 and 6’. With the choice of 0’ = 6, we have 
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log fy (y|@) = F(@) — KL(6(6), f(8)). (77) 


Therefore, subtracting Eq. 77 from Eq. 76, we get 


(KLO(O'), £8) — F(’)) — (KL), £@) — FO) 
= KL(b(6'), b(0)). (78) 


Now we can put the EM algorithm into the alternating-minimization framework. 
Define 


v(b(6'), f(8)) = KL(b(’), f(9)) — F (6). (79) 
We know from Eq. (78) that 
Y(b(8'), £(8)) — V(b(6), f(A) = KL(b(6’), b(A)). (80) 


Therefore, we can say that the M-step of the EM algorithm is to minimize 
w(b(6*), f(@)) over 8 € © to get Ot! and that minimizing Y(b(6), f(6**')) 
gives us 9 = 0*+! again. Because the EM algorithm can be viewed as an alternating 
minimization method, it is also a particular case of the sequential unconstrained 
minimization techniques [13] and of “optimization transfer,” [4]. 

With the choice of 


A(b(6"), b(6)) = KL(b(6), b(@)), 
Eq. (80) becomes 


W(b(6"), f(9)) — V(b), f(9)) = AG(8’), b()), (81) 


which is the three-point property. 

With P = 6(©) and OQ = F(O), the collections of all functions b(@) and 
(0), respectively, we can view the EM algorithm as alternating minimization of 
the function V(p,q), over p € P andgq é€ Q. As we have seen, the three-point 
property holds. What about the four-point property? 

The Kullback-Leibler distance is an example of a jointly convex Bregman 
distance. According to a lemma of Eggermont and LaRiccia [14, 15], the four-point 
property holds for alternating minimization of such distances, using A(p’, p) = 
KL(p’, p), provided that the sets P and Q are closed and convex subsets of R’. 
In the continuous case of the EM algorithm, we are not performing alternating 
minimization on the function KL(b(0), f(6’)), but on KL(b(9), f(6’)) + F(@). 
In the discrete case, whenever Y = h(X), the function F'(@) is always zero, so we 
are performing alternating minimization on the KL distance KL(b(6), f(6’)). In 
[16] the authors consider the problem of minimizing a function of the form 
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A(p.q) = (Pp) + W(q) + De(p.q), (82) 


where # and y are convex and differentiable on R’, D, is a Bregman distance, 
and P = Q is the interior of the domain of g. In [13] it was shown that when Dg 
is jointly convex, the function A(p, q) has the five-point property of [12], which is 
equivalent to the three- and four-point properties taken together. In some particular 
instances, the collection of the functions f(@) is a convex subset of R’, as well, so 
the three- and four-point properties hold. 

As we saw previously, to have V(p",q") converging to d, it is sufficient that 
the five-point property hold. It is conceivable, then, that the five-point property may 
hold for Bregman distances under somewhat more general conditions than those 
employed in the Eggermont-LaRiccia Lemma. 

The five-point property for the EM case is the following: 


KL(b(8), f(6")) — KL@@), FO") 
> (KLO(6"), (0) — FG) — (KLE). f) — FO). (83) 


12 Sums of Independent Poisson Random Variables 


The EM is often used with aggregated data. The case of sums of independent 
Poisson random variables is particularly important. 


Poisson Sums 


Let X,,...,Xy be independent Poisson random variables with expected value 
E(Xn;) = Ay. Let X be the random vector with X, as its entries, 1 the vector 
whose entries are the A,, andA+ = yy An. Then the probability function for X 
is 


N N 


Fx (xl) = |] Ave expan) /xn! = exp(—A4) |] Arr /an! (84) 


n=1 n=1 


Now let Y = bos Xn. Then, the probability function for Y is 


n=1 


II 


Prob(Y = y) = Prob(X; +...+ Xv = y) (85) 


N 
Y> exp(—A4) [Az /an! (86) 


Xp+.xXy=y n=1 


II 
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As we shall see shortly, we have 


N 
Y> exp(—A4) |] Ar” /xn! = exp(-a4)a3/y!. (87) 
n=1 


xi+.xN=y 


Therefore, Y is a Poisson random variable with E(Y) = Ax. 
When we observe an instance of Y, we can consider the conditional distribution 
Sxyy ly, A) of {X1,..., Xw}, subject to y = X, +... + Xy. We have 


fry (ly) = tay" Pn a es (88) 


i!...xy! At At 


This is a multinomial distribution. 
Given y and J, the conditional expected value of X,, is then 


E(Xnly,A) = yan/A. 


To see why this is true, consider the marginal conditional distribution fy,)y (xi|y, A) 
of X;, conditioned on y and A, which we obtain by holding x; fixed and summing 
over the remaining variables. We have 


Sxijy rly.) = y! (= ley 


xil(y — x)! \A4 A+ 
> (y S x1)! I (= i 
! ! / , 
Xo+...+xXy=y-X] ene n=2 a 
where 
M, = AeA. 


As we shall show shortly, 


= N Xp 
~ SSG)" =4 


xo+...+xN=y—-x] n=2 


so that 


fava) ae BY EY 


The random variable X, is equivalent to the random number of heads showing in 
y flips of a coin, with the probability of heads given by 4,/A+. Consequently, the 
conditional expected value of X; is yA, /A+, as claimed. In the next subsection we 
look more closely at the multinomial distribution. 
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The Multinomial Distribution 


When we expand the quantity (a; + ... + ay)”, we obtain a sum of terms, each 
having the form a{!...a,, with x; +... + xy = y. How many terms of the 
same form are there? There are N variables a,,. We are to use x, of the a,, for each 
n=1,...,N,toget y = x; +...+ xy factors. Imagine y blank spaces, each to 
be filled in by a variable as we do the selection. We select x; of these blanks and 
mark them a;. We can do that in () ways. We then select x2 of the remaining blank 
yT—*X1 
x2 


spaces and enter a2 in them; we can do this in ( ) ways. Continuing in this way, 


we find that we can select the N factor types in 
a ee) (89) 
x1 x2 XN-1 


y! (¥=Qitee tan)! _ y! 
xy — x1)! xnav — OG +... + xn-1))! oxy! xw! 


ways or in 


(90) 


This tells us in how many different sequences the factor variables can be selected. 
Applying this, we get the multinomial theorem: 


y! x x 
Pi an N 
@it...tavy’= > Foyt a (91) 
xj +...+xy=y 
Select dy = A,/A+. Then, eee 
i=2=( a ae ae 92 
a i) (92) 
! Ki x2 Any 
=- > —-— la) el) (93) 
xt+...4+xN=y lise tN ae + 


From this we get 


N 
y exp(—A+) I] Aj" [Xy! = exp(—A+)A4_/y! : (94) 


Xy+..xy=y n=1 


13 Poisson Sums in Emission Tomography 


Sums of Poisson random variables and the problem of complete versus incomplete 
data arise in single-photon computed emission tomography (SPECT) [17]. 
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The SPECT Reconstruction Problem 


In their 1976 paper, Rockmore and Makovski [18] suggested that the problem of 
reconstructing a tomographic image be viewed as statistical parameter estimation. 
Shepp and Vardi [19] expanded on this idea and suggested that the EM algorithm 
discussed by Dempster, Laird, and Rubin [11] should be used for the reconstruction. 
The region of interest within the body of the patient is discretized into J pixels (or 
voxels), with A; > 0 the unknown amount of radionuclide within the jth pixel; we 
assume that A; is also the expected number of photons emitted from the jth pixel 
during the scanning time. Emitted photons are detected at any one of J detectors 
outside the body, with y; > 0 the photon count at the ith detector. The probability 
that a photon emitted at the jth pixel will be detected at the ith detector is Pj;, 
which we assume is known; the overall probability of detecting a photon emitted 
from the jth pixelis s; = ey Pi > 0. 


The Preferred Data 

For each i and j, the random variable X;; is the number of photons emitted from 
the jth pixel and detected at the ith detector; the X;; are assumed to be independent 
and P;;A ;-Poisson. With x;; a realization of X;;, the vector x with components x;; 
is our preferred data. The pdf for this preferred data is a probability vector, with 


lod 
fx (xd) =] ] [exp 2 (yay) /xy! - (95) 
i=1j=1 
Given an estimate A* of the vector A and the restriction that Y; = a X;;, the 
random variables X;,,...,X;; have the multinomial distribution 
J 
yi! ( Pijhj ie 
Prob(x;1,...,Xi7) = . 
rob(x;1 xis) acca Ah (PA); 


Therefore, the conditional expected value of X;;, given y and AP is 
E(Xily.A5 = ak Pi (7), 

POMP AR); 
and the conditional expected value of the random variable 


rod 
log fx(X|A) = Yi PiA;) + Xi; log(P;;A;) + constants 
i=1 j=l 
becomes 


Fi 


I 
E(log fx(X|A)ly,a5) = S030 (rian +A Pi Cane, le(P:2), 


i=l j=1 
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omitting terms that do not involve the parameter vector A. In the EM algorithm, we 
obtain the next estimate A+! by maximizing E(log fy(X|A)|y, A). 
The log likelihood function for the preferred data X (omitting constants) is 


I J 
LL») = > > (= Puy + Xi log(PijA))). (96) 


i=1 j=l 
Of course, we do not have the complete data. 


The Incomplete Data 
What we do have are the y;, values of the random variables 


J 
Lay (97) 


j=l 


this is the given data. These random variables are also independent and (PA);- 
Poisson, where 


J 
(PA); = D> PyAj. 


j=l 
The log likelihood function for the given data is 


I 


LLy(A) = D> (= (PA; + yi log((PA)))). (98) 


i=1 


Maximizing LL, (A) in Eq. (96) is easy, while maximizing LL, (A) in Eq. (98) is 
harder and requires an iterative method. 

The EM algorithm involves two steps: in the E-step we compute the conditional 
expected value of L_L,.(A), conditioned on the data vector y and the current estimate 
A* of A; in the M-step we maximize this conditional expected value to get the next 
A*+1. Putting these two steps together, we have the following EMML iteration: 


rg 
k+l _ yk 25, Ji 
i=1 : 


For any positive starting vector A°, the sequence {A*} converges to a maximizer of 
LL, (A), over all nonnegative vectors A. 

Note that because we are dealing with finite probability vectors in this example, 
it is a simple matter to conclude that 


frold= D> fra). (100) 


xeh—l{y} 
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Using the KL Distance 


In this subsection we assume, for notational convenience, that the system y = PA 
has been normalized so that s; = 1 for each j. Maximizing E (log fy(X|A)|y, A") 
is equivalent to minimizing KL(r(A*),q(A)), where r(A) and q(A) are I by J 
arrays with entries 


ra) =A; Pilea) 


and 
q)ij = A; Pij. 


In terms of our previous notation, we identify r(A) with b(0) and q(A) with f(@). 
The set F() of all f(@) is now a convex set and the four-point property of [12] 
holds. The iterative step of the EMML algorithm is then 


I 
k+1 _ yk Ji 
ay Pupp; (101) 


i=l 


The sequence {A‘} converges to a maximizer A y,, of the likelihood for any positive 
starting vector. 

As we noted previously, before we can discuss the possible convergence of the 
sequence {A} of parameter vectors to a maximizer of the likelihood, it is necessary 
to have a notion of convergence in the parameter space. For the problem in this 
section, the parameter vectors A are nonnegative. Proof of convergence of the 
sequence {A‘} depends heavily on the following [20]: 


KL(y, PA*) — KL(y, PA**)) = KL(r(), rAa**) + KL, 125); (102) 
and 


KLO gid) = EL AY Ry PD) = BL, Pe). (AO) 


14 Nonnegative Solutions for Linear Equations 


Any likelihood maximizer A yz is also a nonnegative minimizer of the KL distance 
KL(y, PA), so the EMML algorithm can be thought of, more generally, as a method 
for finding a nonnegative solution (or approximate solution) for a system y = PA 
of linear equations in which y; > 0 and P;; = 0 for all indices. This will be helpful 
when we consider mixture problems. 
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The General Case 


Suppose we want a nonnegative solution x for a system Ax = b of real equations; 
unless b is positive and A has only nonnegative entries, we cannot use the EMML 
algorithm directly. We may, however, be able to transform Ax = b to PA = y. 

Suppose that by rescaling the equations in Ax = b, we can make c; = 
ao Aj; > 0, foreach j = 1,...,J,andb, = ya b; > 0. Now replace Aj; 
with G;; = Aj;/c;,and x; withz; = c;x;;thenGz = Ax = band yy Gi = 1, 
for all 7. We also know now that by = z; > 0, so z+ is now known. 

Let U and u be the matrix and column vector whose entries are all one, 
respectively, and let tf > O be large enough so that all the entries of B= G+tU 
and (tz4+)u are positive. Now 


Br= Gz (tz )u = b + (tz4)u. 
We then solve Bz = b + (tz+)u for z. It follows that Ax = Gz = b and x > 0. 
Finally, we let P = B,A =z, and y = b + (tbi)u. 
Regularization 
It is often the case, as in tomography, that the entries of the vector y are obtained 
by measurements and are therefore noisy. Finding an exact solution of y = PA 
or even minimizing KL(y, PA) may not be advisable in such cases. To obtain an 


approximate solution that is relatively insensitive to the noise in y,* we regularize. 
One way to do that is to minimize not KL(y, PA), but 


F,(A) = (1—a)KL(y, PA) + aKL(p, A), (104) 


where a € (0, 1) and p > Oisa prior estimate of the desired 1. The iterative step of 
the regularized EMML algorithm is now 


I 
ee kl i ; 
ie = (1-a)(aks; > Pu Bqw,) + Ps: (105) 
i=1 : 
As was shown in [20], the sequence {A} converges to a minimizer of F,,(A). 


Acceleration 


When the system y = PA is large, the EMML algorithm can be slow to converge. 
One method that has been used to accelerate convergence to a solution is the use of 
block iteration [21-23]. 
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We begin by writing the index set {i = 1,2,...,/7} as the (not necessarily 
disjoint) union of B,,n = 1,2,...,N. Of particular interest is the row-action 
EMML, obtained by letting each block be a singleton. At each step of the iteration, 
we employ only those equations whose index is a member of the current block. We 
then cycle through the blocks. 

An obvious way to impose blocks would seem to be to modify the EMML 
iteration as follows: 


kt] _ 4k .-l Ji 
a a Pi TB, (106) 


i€Bn 


where 


Sn j = ) Pi;. 


i€By 


This does not work, though. 

Let H; = {z => 0|(Pz); = y;}. Note that for a fixed x > 0, we cannot calculate 
in closed form the vector z € H; that minimizes K L(z, A). However, the vector 
z= 2 in H; that minimizes the weighted KL distance 


J 
>> Py KL, a4) 
j=l 

is given by 


—_) —ie 107 
gj J (PAK); ( ) 


The iterative step of the EMML algorithm can then be interpreted as saying that 
A*+! is a weighted arithmetic mean of the z’; that is, 


I 
i = i) Paz. (108) 


i=1 


This suggests a different form for a block-iterative version of the EMML. 
For k = 0,1,..., andn = n(k) = k(mod N) + 1, let 
k - k -1,k Ji 
a = (1—my'spj)Ah + mz'At 2 iD, (109) 
i€B, v 

where m, = max; S,;. This is the rescaled block-iterative EMML (RBI-EMML) 

algorithm. The sequence {A*} converges to a nonnegative solution of the system 

y = PA, for any choice of blocks, whenever the system has a nonnegative 
solution [21]. 
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When each block is a singleton, that is, B, = B; = {i},fori = 1,2,...,. = N, 
the RBI-EMML becomes the EMART algorithm, with the iterative step 
Ji 


k = k rr | 
where m; = max; P;; > 0. It is interesting to compare the EMART algorithm with 
the multiplicative algebraic reconstruction technique (MART) [24], which has the 
iterative step 


k+l _ yk Ji Py [mi 

os =i (aw) (111) 

so that 

1—Pij /mi : Pij /Mi 
k+l _ (yk J kk Si ij 
iin (45) (25 Ge, , ate) 
or 

log K+! = (1 — my! P,) log Ak + m7" P;; log (a4 ma (113) 


The difference between the MART and the EMART is then the difference between 
a geometric mean and an arithmetic mean. 

The simultaneous MART (SMART) is analogous to the EMML and uses all the 
equations at each step [20, 25, 26]. The iterative step for the SMART is 


I 

k+1 _ yk -1 Ji 

ai = AK exp (sj y Pi los Gaz): (114) 
i=l . 


Block-iterative versions of the MART (RBI-SMART) have been considered by [27] 
and [21]. When y = PA has nonnegative solutions, the RBI-SMART sequence 
converges to the nonnegative solution of y = PA for which the cross-entropy 
KL(A,A°) is minimized. When there are no nonnegative solutions of y = PA, 
the SMART converges to the nonnegative minimizer of KL(PA,y) for which 
KL(A,A°) is minimized [28]. 


Using Prior Bounds on A 


The EMML algorithm finds an approximate nonnegative solution of y = PA. In 
some applications it is helpful to be able to incorporate upper and lower bounds on 
the A [29]. 

The SMART, EMML, MART, and EMART methods are based on the Kullback- 
Leibler distance between nonnegative vectors. To impose more general constraints 


EM Algorithms from a Non-stochastic Perspective 421 


on the entries of A, we derive algorithms based on shifted KL distances, also called 
Fermi-Dirac generalized entropies. 

For a fixed real vector u, the shifted KL distance K L(x — u,z — u) is defined 
for vectors x and z having x; > u; and z; > uj. Similarly, the shifted distance 
KL(v — x,v — z) applies only to those vectors x and z for which x; < v; and 
Zj <v;.Foru; < v;, the combined distance 


KL(x —u,z—u) + KL(v—x,v—z) 


is restricted to those x and z whose entries x; and z; lie in the interval [w;, v;]. Our 
objective is to mimic the derivation of the SMART and EMML methods, replacing 
KL distances with shifted KL distances, to obtain algorithms that enforce the 
constraints u; < A; < vj, foreach j. The algorithms that result are the ABMART 
and ABEMML block-iterative methods. These algorithms were originally presented 
in [30], in which the vectors u and v were called a and b, hence the names of the 
algorithms. We shall assume that the entries of the matrix P are nonnegative. We 
shall denote by B,, n = 1,..., N a partition of the index set {i = 1,..., 7} into 
blocks. Fork = 0,1,...letn = n(k) = k(mod N) + 1. 


The ABMART Algorithm 
We assume that (Pu); < y; < (Pv); and seek a solution of PA = y with u; < 
A; <v;, for each j. The algorithm begins with an initial vector A° satisfying u ix 
as < v;, for each j. Having calculated A*, we take 
MT = oy; + af )uj, (115) 
with n = n(k), 
,_ gmap” 
od en PCS PT (116) 
+; ap 
(AK —u;) 
ck= 1 (117) 
(vj - Xi) 
and 
i; — (Pu)i)((Pv); — (PAY); 
jk = Giz (Puy = (PA) Aas 


1 (Pv) — yi)((PAX); — (Pu)i)’ 


where TI’ denotes the product over those indices 7 in B,(,). Notice that at each step 
of the iteration, A“ is a convex combination of the endpoints u; and v;, so that A 
always lies in the interval [u;, v;]. 

We have the following theorem concerning the convergence of the ABMART 
algorithm: 


422 C. Byrne 


Theorem 1. /f there is a solution of the system PX = y that satisfies the constraints 
uj < Aj; < vj; for each j, then, for any N and any choice of the blocks By, the 
ABMART sequence converges to that constrained solution of PA = y for which the 
Fermi-Dirac generalized entropic distance from X to d°, given by 


KLQA=u4,4°=0) + KLW=1,0=— 9’), 


is minimized. If there is no constrained solution of PA = y, then, for N = 1, the 
ABMART sequence converges to the minimizer of 


KL(PA— Pu,y — Pu) + KL(Pv— Pd, Pv—y) 
for which 
RLQ=00° =a + ELG—1.9=52°) 
is minimized. 
The proof is in [30]. 
The ABEMML Algorithm 


We make the same assumptions as previously. The iterative step of the ABEMML 
algorithm is 


Ae = aky; + (1—ai)u;, (119) 

where 
gaye ds (120) 
Pots (2 
KE = (yj AA) fF, (122) 
dk = yk + Bh (123) 

i — (Pu); 
ek = (: oa rs] ~ dX Pil in Oe). (124) 
and 


a _ 2 7 (Pv)i — yi 
i= (: 2 rs] + >> Py (es = ain) (125) 


ieB, i€By 


The following theorem concerns the convergence of the ABEMML algorithm: 
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Theorem 2. [f there is a solution of the system PX = y that satisfies the constraints 
uj < Aj < vj; for each j, then, for any N and any choice of the blocks By, the 
ABEMML sequence converges to such a constrained solution of PX = y. If there 
is no constrained solution of PA = y, then, for N = 1, the ABEMML sequence 
converges to a constrained minimizer of 


KL(y — Pu, PA— Pu) + KL(Pv—y, Pv— PA). 


The proof is found in [30]. In contrast to the ABMART theorem, this is all we can 
say about the limits of the ABEMML sequences. 


15 Finite Mixture Problems 


Estimating the combining proportions in probabilistic mixture problems shows 
that there are meaningful examples of our acceptable-data model, and provides 
important applications of likelihood maximization. 


Mixtures 


We say that a random vector V taking values in R? is a finite mixture [31,32] if 
there are probability density functions or probabilities f; and numbers 6; > 0, for 
j =1,...,J, such that the probability density function or probability function for 
V has the form 


J 
fv 8) = 956; f0). (126) 
j=l 


J 


for some choice of the 6; > 0 with ae 


without loss of generality, that D = 1. 


0; = 1. As previously, we shall assume, 


The Likelihood Function 


The data are N realizations of the random variable V , denoted v,, forn = 1,...,N, 
and the given data is the vector y = (vy,...,vy). The column vector 9 = 
(6,,...,0,)7 is the generic parameter vector of mixture combining proportions. 
The likelihood function is 


N 


Ly(9) = T] (fin) +... + 8 fa (0n)). 127) 


n=1 


Then the log likelihood function is 
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N 
LLy(6) = log (1 filtn) +--+ 8s f1(n)). 


n=1 


With wu the column vector with entries u, = 1/N, and P the matrix with entries 
Pj = ti (Vn), we define 


N N 
eS 2 Pa = 2, fi») 


Maximizing LL, (@) is equivalent to minimizing 


J 
F(6) = KL(u, P@) + Y (1 = 5;)6;. (128) 


j=l 


A Motivating Illustration 


To motivate such mixture problems, we imagine that each data value is generated 
by first selecting one value of j, with probability 6;, and then selecting a realization 
of a random variable governed by /;(v). For example, there could be J bowls 
of colored marbles, and we randomly select a bowl, and then randomly select a 
marble within the selected bowl. For each n the number v, is the numerical code 
for the color of the nth marble drawn. In this illustration we are using a mixture of 
probability functions, but we could have used probability density functions. 


The Acceptable Data 


We approach the mixture problem by creating acceptable data. We imagine that 
we could have obtained x, = j,, form = 1,...,N, where the selection of v, 
is governed by the function /;,(v). In the bowls example, j,, is the number of the 
bowl from which the nth marble is drawn. The acceptable-data random vector is 
X = (X1,..., Xn), where the X,, are independent random variables taking values 
in the set {7 = 1,..., J}. The value j, is one realization of X,,. Since our objective 
is to estimate the true 6;, the values v, are now irrelevant. Our ML estimate of 
the true 9; is simply the proportion of times j = j,. Given a realization x of X, 
the conditional pdf or pf of Y does not involve the mixing proportions, so X is 
acceptable. Notice also that it is not possible to calculate the entries of y from those 
of x; the model Y = h(X) does not hold. 
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The Mix-EM Algorithm 


425 


Using this acceptable data, we derive the EM algorithm, which we call the Mix-EM 


algorithm. 


With N; denoting the number of times the value j occurs as an entry of x, the 


likelihood function for X is 
u N 
L (9) = fx(xl0) =] 97", 
j=l 
and the log likelihood is 
LLx(9) = log Ly(8) = )° Nj log 6;. 


j=l 


Then 


J 
E(log Lx (9)|y, 0") = >> E(Njly, 0°) log 6;. 
j=l 
To simplify the calculations in the E-step, we rewrite L.L,.(@) as 
N oJ 
LLx(9) = ¥- > Xnj log 6;, 


n=1 j=1 


where X,; = lif j = j, and zero otherwise. Then we have 


ok fir, 
E(Xnj\y, 6") = prob (has = lly, 0") 2a ad Si ) 


The function E(LL,(6)|y, 0*) becomes 


N J ok i 
E(LLa Oly. 08) = SV Ty e8 


vA n 
Qk 
n=1j=1 Vn| 


Fn) 


(129) 


(130) 


(131) 


(132) 


(133) 


(134) 


Maximizing with respect to 0, we get the iterative step of the Mix-EM algorithm: 


k+l fi Wn) 
3 ty fi 


(135) 
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We know from our previous discussions that since the preferred data X is 
acceptable, likelihood is non-decreasing for this algorithm. We shall go further now 
and show that the sequence of probability vectors {9} converges to a maximizer of 
the likelihood. 


Convergence of the Mix-EM Algorithm 


As we noted earlier, maximizing the likelihood in the mixture case is equivalent to 
minimizing 
J 


F(6) = KL(u, P#™) + Y= 35))6;, 


j=l 


over probability vectors @. It is easily shown that if 6 minimizes F (0) over all 

nonnegative vectors @, then 6 isa probability vector. Therefore, we can obtain the 

maximum likelihood estimate of 6 by minimizing F(@) over nonnegative vectors 6. 
The following theorem is found in [33]. 


Theorem 3. Let u be any positive vector, P any nonnegative matrix with s; > 0 
for each j, and 


J 
F(9) = KL(u, P60) + ) > Bj KL(yj.9)). 


j=l 


Ifs; + Bj > 0,a; = s;/(s; + Bj), and Bjy; = 90, for all j, then the iterative 
sequence given by 


N 
k+l _ —1gk Un 
0; = ajS; ar ( Pad pgR,) + ays (136) 


n= 
converges to a nonnegative minimizer of F(@). 


With the choices u, = 1/N, y; = 0, and 8; = 1 —s,, the iteration in Eq. (136) 
becomes that of the Mix-EM algorithm. Therefore, the sequence {0*} converges to 
the maximum likelihood estimate of the mixing proportions. 


16 More on Convergence 
There is a mistake in the proof of convergence given in [11]. Wu [34] and Boyles 


[35] attempted to repair the error but also gave examples in which the EM algorithm 
failed to converge to a global maximizer of likelihood. In Chap.3 of the book 
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by McLachlan and Krishnan [1], we find the basic theory of the EM algorithm, 
including available results on convergence and the rate of convergence. Because 
many authors rely on Eq. (14), it is not clear that these results are valid in the 
generality in which they are presented. There appears to be no single convergence 
theorem that is relied on universally; each application seems to require its own proof 
of convergence. When the use of the EM algorithm was suggested for SPECT and 
PET, it was necessary to prove convergence of the resulting iterative algorithm in 
Eq. (99), as was eventually achieved in a sequence of papers [19, 36-38], and [20]. 
When the EM algorithm was applied to list-mode data in SPECT and PET [39- 
41], the resulting algorithm differed slightly from that in Eq. (99) and a proof of 
convergence was provided in [33]. The convergence theorem in [33] also establishes 
the convergence of the iteration in Eq. (135) to the maximum-likelihood estimate of 
the mixing proportions. 


17 Open Questions 


As we have seen, the conventional formulation of the EM algorithm presents 
difficulties when probability density functions are involved. We have shown here 
that the use of acceptable preferred data can be helpful in resolving this issue, but 
other ways may also be useful. 

Proving convergence of the sequence {@*} appears to involve the selection of an 
appropriate topology for the parameter space ©. While it is common to assume that 
© is a subset of Euclidean space and that the usual norm should be used to define 
distance, it may be helpful to tailor the metric to the nature of the parameters. In 
the case of Poisson sums, for example, the parameters are nonnegative vectors, and 
we found that the cross-entropy distance is more appropriate. Even so, additional 
assumptions appear necessary before convergence of the {9*} can be established. 
To simplify the analysis, it is often assumed that cluster points of the sequence lie 
in the interior of the set ©, which is not a realistic assumption in some applications. 

It may be wise to consider, instead, convergence of the functions fy (x|0*) 
or maybe even to identify the parameters 6 with the functions fy (x|@). Proving 
convergence to L (0) of the likelihood values L,(6*) is also an option. 


18 Conclusion 


Difficulties with the conventional formulation of the EM algorithm in the continuous 
case of probability density functions (pdf) have prompted us to adopt a new 
definition, that of acceptable data. As we have shown, this model can be helpful 
in generating EM algorithms in a variety of situations. For the discrete case of 
probability functions (pf), the conventional approach remains satisfactory. In both 
cases, the two steps of the EM algorithm can be viewed as alternating minimization 
of the Kullback-Leibler distance between two sets of parameterized pf or pdf, along 
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the lines investigated by Csiszar and Tusnady [12]. In order to use the full power of 
their theory, however, we need the sets to be closed and convex. This does occur in 
the important special case of sums of independent Poisson random variables, but is 
not generally the case. 


Acknowledgments I wish to thank Professor Paul Eggermont of the University of Delaware for 
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Abstract 

This chapter deals with iterative methods for nonlinear ill-posed problems. We 

present gradient and Newton type methods as well as nonstandard iterative algo- 

rithms such as Kaczmarz, expectation maximization, and Bregman iterations. 
Our intention here is to cite convergence results in the sense of regularization 

and to provide further references to the literature. 


1 Introduction 


This chapter will be devoted to the iterative solution of inverse problems formulated 
as nonlinear operator equations 


F(x)=y, (1) 


where F : D(F) > Y with domain D(F) C 4. The exposition will be mainly 
restricted to the case of XY and Y being Hilbert spaces with inner products ( - , - ) 
and norms || - ||. Some references for the Banach space case will be given. 

We will assume attainability of the exact data y in a ball B, (x0), i-e., the equation 
F(x) = y is solvable in B,(xo). The element xo is an initial guess which may 
incorporate a-priori knowledge of an exact solution. 

The actually available data y° will in practice usually be contaminated with noise 
for which we here use a deterministic model, i.e., 


ly’ - yl <8, (2) 
where the noise level 5 is assumed to be known. For a convergence analysis with 


stochastic noise, see the references in section “Further Literature on Gauss—-Newton 
Type Methods”. 


2 Preliminaries 

Conditions on F 

For the proofs of well-definedness and local convergence of the iterative methods 
considered here we need several conditions on the operator F’. Basically, we 
inductively show that the iterates remain in a neighborhood of the initial guess. 


Hence, to guarantee applicability of the forward operator to these iterates, we 
assume that 


Bop(x0)  D(F) (3) 


for some p > 0. 
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Moreover, we need that F is continuously Fréchet-differentiable, that || F’(x)|| 
is uniformly bounded with respect to x € Bz,(xo), and that problem (1) is properly 
scaled, i.e., certain parameters occurring in the iterative methods have to be chosen 
appropriately in dependence of this uniform bound. 

The assumption that F’ is Lipschitz continuous, 


IF @)- F(x) < LI|k—xll, x, ¥ © Bop(%o), (4) 


that is often used to show convergence of iterative methods for well-posed problems, 
implies that 


| FQ) — F(x) — F'(x)\(%—x)|| <cl]¥ —xI1?, x, ¥ © Bop(Xo). (5) 


However, this Taylor remainder estimate is too weak for the ill-posed situation 
unless the solution is sufficiently smooth (see, e.g., case (11) in Theorem 9 below). 
An assumption on F that can often be found in the literature on nonlinear ill-posed 
problems is the tangential cone condition 


| F(x) — F@) — F°@)e — x) <0 


|F@)-F@)Il, n< x,X € Byp(xo) S D(F), (6) 


1 
2? 
which implies that 


1 . 7 1 e 
TI F’'@G -*Il < IF@ — FOI = — IF’ WE -ll 
l+n 7 
for all x, X € Bzp)(xo). One can even prove (see [70, Proposition 2.1]). 


Proposition 1. Let p,¢ > 0 be such that 
| FQ) — F&) — F’(x)(« — X)|| < e@, x) 
F(x) — FO), x, x € Bp(xo) S D(F), 
for some c(x, X) = 0, where c(x,%) < Lif ||x —£l| <«. 
(i) Then for all x € B,(x0) 
My := {X € Bp(xo) : F(X) = F(x)} =x + N(F(x)) 9 Bp(x0) 
and N(F'(x)) = N(F'(%)) for all & € My. Moreover, 


N(F'(x)) D {t(i—x) 1 FE M,,t € R}, 


where instead of > equality holds if x € By(xo). 


434 M. Burger et al. 


(ii) If F(x) = y is solvable in B,(Xo), then a unique X9-minimum-norm solution 
exists. It is characterized as the solution x‘ of F(x) = y in B,(x0) satisfying 
the condition 


=e NO'G')) Ce. (7) 


If F(x) = y is solvable in B,(xo) but a condition like (6) is not satisfied, then at 
least existence (but no uniqueness) of an x9-minimum-norm solution is guaranteed 
provided that F' is weakly sequentially closed (see [36, Chapter 10]). 

For the proofs of convergence rates one even needs stronger conditions on F’ 
than condition (6). 


Source Conditions 


It is well known by now that the convergence of regularized solutions can be 
arbitrarily slow. Rates can only be proven if the exact solution x’ satisfies some 
regularity assumptions, so-called source conditions. They are usually of Hélder- 
type, i.e., 


xbox = (FON FO )y, ve N(F (xt) (8) 


for some exponent jz > 0. Due to typical smoothing properties of the linearized 
forward operator F’(x"), they can be interpreted as smoothness assumptions on the 
initial error xt — xo. 

Since (8) is usually too strong for severely ill-posed problems, logarithmic source 
conditions, i.e., 


xt—xg = Fr Gyr ay, w>0, vEeN(F'(x*))4, 


(9) 
FEA) = (Cincy), er > e?, 
have been considered by Hohage [56] (cf. [30, Theorem 2.7] for Landweber 


iteration, [54] for the iteratively regularized Gauss-Newton method IRGNM, and 
[55] for generalized IRGNM). 


Stopping Rules 


In the context of ill-posed problems it is essential to stop iterative solution methods 
according to an appropriate rule to avoid an unbounded growth of the propagated 
noise. There are two possibilities, either a-priori rules or a-posteriori rules. A-priori 
tules [see, e.g., (58)] are computationally very effective. However, the disadvantage 
is that one has to know the smoothness index jz in (8) or (9) explicitly. 
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This is avoided in a-posteriori stopping rules. The most well-known a-posteriori 
criterion is the so-called discrepancy principle, i.e., the iteration is stopped after 
kx = kx (8, y°) steps with 


lly’ — FOR OI <8 < I? — PQQ, OSk <ks, (10) 


where t > 1. 

The Lepskii type balancing principle is an interesting alternative a-posteriori rule, 
see [6, 8] and (62) below in the context of iterative methods. 

If the noise level 6 in (2) is unknown, heuristic stopping rules such as the 
quasioptimality principle, the Hanke—Raus rule, or the L-curve criterion still lead 
to convergence under certain structural assumptions on the noise, see [7, 74,75, 89]. 


3 Gradient Methods 


One way to derive iterative regularization methods is to apply gradient methods to 
the minimization problem 


min $ || F(x) —yll? over D(F). 


Since the negative gradient of this functional is given by F’(x)*(y — F(x)) and 
taking into account that only noisy data y® are available, this yields methods of the 
form 


xb, = xb + of F’(xt)*(y' — F(x), (11) 


where xe = Xo is an initial guess of the exact solution. Choosing the factor a} in 
a special way we obtain well-known methods like Landweber iteration, the steepest 
descent method, and the minimal error method. 


Nonlinear Landweber Iteration 


If one chooses a) = o to be constant, one obtains Landweber iteration. As already 


mentioned in the introduction of this chapter, well-definedness and convergence can 
only be proven if problem (1) is properly scaled. Without loss of generality we may 
assume that o> = | and that 


IF’) <1, x € Byp(%o) C D(F). (12) 
The nonlinear Landweber iteration is then given as the method 


x8, = x8 + F'(x8)*Gy' — Fs), ok ENo. (13) 
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We want to emphasize that for fixed iteration index k the iterate x depends 
continuously on the data y®, since xe is the result of a combination of continuous 
operations. 

The results on convergence and convergence rates for this method presented here 
were established in [49] (see also [70]). To begin with, we formulate the following 
monotonicity property that gives us a clue how to choose the number t in the 
stopping rule (10) (see [70, Proposition 2.2]). 


Proposition 2. Assume that the conditions (12) and (6) hold and that the equation 
F(x) = y has a solution x, € B,(Xxo). If x € B,(xx), a sufficient condition for 
Mig to be a better approximation of X* than x, is that 


1+7 
3 3 
—F > 2—— 6. 
ly’ - FOI > 275, 
Moreover, it then holds that ce ea € By(Xx) C Bop (Xo). 
In view of this proposition, the number t in the stopping rule (10) should be 
chosen as 


1+ n 


T = 2 —_— 
1-27 


with 7 as in (6). To be able to prove that the stopping index k, in (10) is finite 
and hence well defined it turns out that t has to be chosen slightly larger (see [70, 
Corollary 2.3]), i.e., 


>2; (14) 
u) 


Corollary 1. Let the assumptions of Proposition 2 hold and let kx be chosen 
according to the stopping rule (10), (14). Then 


kl 
kyl cby = Ma Fg? = r xo — Xx||. 
(78) PB I’ — FOI? < Go aap ro el 
In particular, if y° = y (i.e. if 5 = 0), then 
CO 
do lly — F@x)I? < 00. (15) 


k=0 


Note that (15) implies that if Landweber iteration is run with precise data y = y®, 
then the residual norms of the iterates tend to zero as k — oo. That is, if the iteration 
converges, then the limit is necessarily a solution of F(x) = y. The following 
convergence result holds (see [70, Theorem 2.4]): 
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Fig. 1 The sketch shows the 
initial element xo, the 
Xo-minimum-norm solution 
x1, the subset 

xt + .N(F’(x")) and in bold 
the region, where the limit of 
the iterates x; can be 


at + N(F'(at)) 


Theorem 1. Assume that the conditions (12) and (6) hold and that the equation 
F(x) = y is solvable in B,(xo). Then the nonlinear Landweber iteration applied to 
exact data y converges to a solution of F(x) = y. If N(F'(x')) C N(F'(x)) for 
all x € B,(x"), then x, converges to x‘ as k — ov. 


We emphasize that, in general, the limit of the Landweber iterates is no xo- 
minimum-norm solution. However, since the monotonicity result of Proposition 2 
holds for every solution, the limit of x; has to be at least close to x7. As can be seen 
below, it has to be the closer the larger p can be chosen (Fig. 1). 

It is well known that, if y> does not belong to the range of F, then the iterates 
xe of (13) cannot converge but still allow a stable approximation of a solution of 
F(x) = y provided the iteration is stopped after k, steps. The next result shows 
that the stopping rule (10), (14) renders the Landweber iteration a regularization 
method (see [70, Theorem 2.6]): 


Theorem 2. Let the assumptions of Theorem 1 hold and let kx = kx(5, y°) be 
chosen according to the stopping rule (10), (14). Then the Landweber iterates ae 


converge to a solution of F(x) = y. IfN(F'(x")) C N(F'(x)) forall x € B(x"), 
5 


then x; converges to xt as& > 0. 

To obtain convergence rates the exact solution has to satisfy some source 
conditions. Moreover, one has to guarantee that the iterates remain in R(F’(x‘)*). 
In [49] rates were proven under the additional assumption that F' satisfies 


F'(x) = Rx F'(x') and ||R,—J|| <c|lx—x"ll, X € Byp(xo), 


where {Rx : x € Bap(xo)} is a family of bounded linear operators R, : Y > Y 
and c is a positive constant. 

Unfortunately, these conditions are not always satisfied (see [49, Example 4.3]). 
Therefore, we consider instead of (13) the following slightly modified iteration 
method, 
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x84, =x! + 68 (x8)*(y? — F(x’), kk ENo, (16) 


where, as above, x4 = Xo is an initial guess, G(x) := G(x, y°), and G is a 
continuous operator mapping D(F) x Y into L(V, Y). The iteration will again be 
stopped according to the discrepancy principle (10). 

To obtain local convergence and convergence rates for this modification we need 
the following assumptions: 


Assumption 9. Let p be a positive number such that Bz)(xo) C D(F). 
(i) The equation F(x) = y has an x9-minimum-norm solution x' in B,(xo). 
(ii) There exist positive constants c,, C2, C3 and linear operators Ré such that for 


allx € B,(x") the following estimates hold: 


IFO) — FQ) — FON @— x) < er lF@)- FO) x27], a7 


G*(x) = R°G*(x"), (18) 
Ro TI <collx —x" I, (19) 
IF’ (xt) — G8 (xt) || < 038. (20) 


(iii) The scaling parameter w in (16) satisfies the condition 
ol|F'O)IP <1. 
Note that, if instead of (17) the slightly stronger condition 


| F(x) — F@) — F°@)@ - x) < ellx- x] 
| F@) — F(X), x, X € Bop(xo) € D(F), (21) 
holds in B2,(xo) for some c > 0, then the unique existence of the xo-minimum-norm 
solution x* follows from Proposition | if F(x) = y is solvable in B, (xo). 
Convergence and convergence rates for the modification above are obtained as 


follows (see [70, Theorems 2.8 and 2.13]): 


Theorem 3. Let Assumption 9 hold and let kx = kx(6, y°) be chosen according to 
the stopping rule (10). 


(i) If \|xo — x"|| is so small and if the parameter t in (10) is so large that 


2m + n3n3 <2 
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and 


= 2(1 + m + ¢3M2 ||x0 — x" ||) 
2—2m — non3 


’ 


where 


m= |lxo— x" | (cr + 21 + 1 |]x0 — x7), 
No = 14+ e|xo — x" I, 


II 


nz = 1+ 2c3|!xo0 — x" |], 


then the modified Landweber iterates xe converge to x' as 8 + 0. 
(ii) If t > 2 and if xt — x9 satisfies (8) with some 0 < ju < 1/2 and ||v|| sufficiently 
small, then it holds that 


ke = O(|\v| FS 7) 
and 
o( Il FI), w< 4, 
(VI). wah 


Note that for the modified Landweber iteration we obtain the same convergence 
rates and the same asymptotical estimate for k, as for linear ill-posed problems 
(compare [36, Theorem 6.5]) if ~ < 1/2 in (8). 

Under the Assumption 9 and according to the theorem above the best possible 
convergence rate is 


3 
xe, — 27 = 


Ine, — x7 = O(/8) 
provided that 44 = 1/2. Even if 4 > 1/2 we cannot improve this rate without an 
additional restriction of the nonlinearity of F. 
We will show for the following parameter estimation problem that the conditions 
of Assumption 9 are satisfied if F’(x) is replaced by a certain operator G(x). 
Example 1. We treat the problem of estimating the diffusion coefficient a in 


—(a(s)u(s)s)s = f(s), s€(0,1), u@)=0=u(1), (22) 


where f € L?; the subscript s denotes derivative with respect to s. 
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In this example, F is defined as the parameter-to-solution mapping 


F : D(F) := {a € H'[0,1] : a(s) >a > 0} > L’(0, 1] 
at> F(a) :=u(a), 
where u(a) is the solution of (22). One can prove that F is Fréchet-differentiable 
(see, e.g., [24]) with 
F'(ayh = A(a)'[(hus(a))s] 
F'(a)*w = —B™[us(a)(A(a)“'w)s], 


where 


A(a) : H7[0, 1] Hg (0, 1] > L7[0, 1] 


ub» A(a)u := —(aus)s 
and 


B:D(B):= {wv € H’[0,1] : W’() = WG) = 0} > L’(0, 1] 
yi By = —w" ty; 
note that B~! is the adjoint of the embedding operator from H '[0, 1] in L7[0, 1]. 
First of all, we show that F satisfies condition (17): let F(a) = u, F(a) = u, 
and w € L?. Noting that (# — u) € H?N i, and that A(a) is one-to-one and onto 
for a,a € D(F) we obtain that 
( F(a) — F(a) — F'(a)(@—a),w) ) 12 
= ( (u = u) — A(a)™ [((a = a)us)s|, WwW ) 722 
= (A(a)(i— u) — (G — a)us)s, Ala) 'w) 12 
= (((@—a)(iis — us))s, A@)'w) 22 
= —((@—a)(t—u)s, (A(a)'W)s ) 12 
= ( F(@) — F(a), (@— a)(A@)'W)s)s ) 12 - 
This together with the fact that ||g||,-0 < V2||g|| 7: and that ||g||L00 < |g’ ||,2 if 
g € H' is such that g(€) = 0 for some & € (0, 1] yields the estimate 
| F(@) — F(a) — F’(ay(@—a) |e 
< sup (F(@)— F(a), (G@—a)(A(@)'W)5)s ) 12 


wll ;2=1 
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<IF@—-F@l sp [|(=*) | 


wll j2=1 a 


2 la(A@ w)slles 


+ [A]. be 
a_\lLe me 
<a'+V2+a! v2 lalla) |F@ - F@| 2 la-alla- (23) 
This implies (17). 


The conditions (18) and (19) are not fulfilled with G°(x) = F’(x). Noting that 
F'(a)*w is the unique solution of the variational problem: for all vy € H! 


((F'(a)*w)s, Vs) 122 + ( F'(a)*w.v) 22 = (u(a), ((A(a)|W)s5¥)5 ) 22 (24) 


we propose to choose G° in (16) as follows: G°(a)*w = G(a, u’)*w is the unique 
solution g of the variational problem 


(gs. ¥s)22 + (8,v)22 = (uv, (A@)'w)sv)s)a2, ve Ht. (25) 

This operator G* obviously satisfies (18), since 
G(a,u’)* = G(a,u’)* RG, a)* 
with 
R(G,a)* = A(a)A(a@)!. 

The condition (19) is satisfied, since one can estimate as in (23) that 
|RG@.a)*-1|| = |A@MA@ "1 || <a + V24+a' V2 [all l@—alln. 
Note that a constant c2 independent from a can be found, since it is assumed that 


a € B,(a). Now we turn to condition (20): using (24) and (25) we obtain similarly 
to (23) the estimate 


|(F(a)* — Ga,u’)*)w||q1 = sup (u(a) — w®, ((A(a)'W)5¥)5 22 


Ivll=l 


<a'+ v24+a! V2 Ial| 7) |le@ — 0 ll [wll - 
This together with F(a’) = u(a*) and ||u® — u(a*)||,2 < 5 implies that 
IF’ @)—Galw) <a d+ v2 +a V2 Ila‘) 5 


and hence (20) holds. 
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Thus, Theorem 3 is applicable, i.e., if @ and t are chosen appropriately, then 
the modified Landweber iterates al (cf. (16)) where k, is chosen according to the 


stopping rule (10) converge to the exact solution a* with the rate O(./8) provided 
that 


a’ — ay = —B'[us(a)(A(a’)w)] 


with ||w|| sufficiently small. Note that this means that 


at'—a,€ H?, (ai — ap);(0) = 0 = (a — ap),(1), 
t_ _ — 1 
7 (a! — ao)ss — (a' — ao) ent / Aoh=a 
us (a) 0 


Basically this means that one has to know all rough parts of at up to H?. But without 
this knowledge one cannot expect to get the rate O(+/8). 


In [49] two other nonlinear problems were treated where conditions (18) and (19) 
are satisfied with G°(x) = F’(x). 


Landweber Iteration in Hilbert Scales 


We have mentioned in the last subsection that for classical Landweber iteration the 
rates cannot be better than O(./6) under the given assumptions. However, better 
rates may be obtained for solutions that satisfy stronger smoothness conditions if 
the iteration is performed in a subspace of VY with a stronger norm. This leads us 
directly to regularization in Hilbert scales. On the other hand for solutions with 
poor smoothness properties the number of iterations may be reduced if the iteration 
is performed in a space with a weaker norm. 

First of all, we shortly repeat the definition of a Hilbert scale: let L be a densely 
defined unbounded self-adjoint strictly positive operator in V. Then (%;)seR denotes 
the Hilbert scale induced by L if 4, is the completion of ()7~- D(L*) with respect 
to the Hilbert space norm ||x||5 := ||Z°x||; obviously, ||x||o = ||x||~ (see [78] or 
(36, Section 8.4] for details). 

The operator F’ (x?)* in (13) will now be replaced by the adjoint of F’ (x?) 
considered as an operator from -¥,; into Y. Usually s > 0, but we will see below 
that there are special cases where a negative choice of s can be advantageous. Since 
by definition of %, this adjoint is given by L~~ F’ (x?)*, (13) is replaced by the 
iteration process 


24a) EOF G" =F GS), ke Ne: (26) 


As in the previous chapter the iteration process is stopped according to the 
discrepancy principle (10). 
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Proofs of convergence and convergence rates for this method can be found in 
[34, 70, 88]. For an approach, where the Hilbert scale is chosen in the space Y, 
see [33]. 

The following basic conditions are needed. 


Assumption 10. 


(i) F : D(F)(C X) => Y is continuous and Fréchet-differentiable in X. 
(ii) F(x) = y has a solution x". 
(iii) ||F’(x")x|| < m||x||-a for all x € X and some a > 0, m > 0. Moreover, the 
extension of F'(x') to X_g is injective. 
(iv) B := F'(x')L~ is such that ||B||v. < 1, where —a < s. If s < 0, F’(x‘) 
has to be replaced by its extension to X;. 


Usually, for the analysis of regularization methods in Hilbert scales a stronger 
condition than (iii) is used, namely (cf., e.g., [88]) 


|F’'(x")xl| ~ |lxl]-a forall x € &, (27) 


where the number a can be interpreted as a degree of ill-posedness of the linearized 
problem in x‘. However, this condition is not always fulfilled. Sometimes one can 
only prove that condition (iii) in Assumption 10 holds. It might also be possible that 
one can prove an estimate from below in a slightly weaker norm (see examples in 
[34]), i-e., 


|F’(x") x] > m||xl|—2 for all x € XY andsome G4 >a,m>0. (28) 
The next proposition sheds more light onto condition (iii) in Assumption 10 and 
(28). The proof follows the lines of [36, Corollary 8.22] noting that the results there 
not only hold for s > 0 but also for s > —a. 
Proposition 3. Let Assumption 10 hold. Then for all v € [0, 1] it holds that 
D((B* BY?) = R((B*B)?) C Xyats) 
(B*B)2x|| < m7 \|x||vats) forall x eX, 
|(B*By 2x > 7" |xlvats) forall, x € D(B*B)-?). 
Note that condition (iii) is equivalent to 
R(F'(x")*) CX, and || F’(x')*wlla <m|lw|| forall wey. 


If in addition condition (28) holds, then for all v € [0, 1] it holds that 
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XyG+s) C R((B*B)?) = D((B*B)?), 
|(B*B)2x]] =m" |xll-v@+s) forall x eX, 
|(B*B)-?x|| < m- |[xll\@4sy forall, x € Ay@4s). 


Note that condition (28) is equivalent to 


Mg C R(F'(x!)*) and || F’(x!)*wl|z = mw 


forall we N(F'(xt)*)+ with F’(xt)*w € 4X3. 


In our convergence analysis the following shifted Hilbert scale will play an 
important role 


&, = D((B* B) 245 L*) equipped with the norm 


sor 


[ll] = [(B*B) 49 L’x| x, 


where a, s, and B are as in Assumption 10. Some properties of this shifted Hilbert 
scale can be found in [70, Proposition 3.3]. 

For the convergence rates analysis we need the following smoothness conditions 
on the solution x* and the Fréchet-derivative of F. 


Assumption 11. 


(i) xo € B,(x*) = {xe Xs x—xt eX A |lx—x'l]o < p} C D(F) for some 
p> 0. 
(ii) lF’'C)-—F’@)ll gy» <c l|xt x2 for all x € B,(x') and some b € [0, a], 
B € (0, 1], and c > 0. 
(iii) xt —xq € X, for some (a—b)/B <u <b+2s, ie, there is an elementv € X 
so that 


L' (xt — x9) = (B*B)%+5y and |lv|lo = [xo — x" lh. 


Condition (iii) is a smoothness condition for the exact solution comparable to 
(8). Usually ¥, is used instead of x. However, these conditions are equivalent if 
(27) holds. 

For the proof of the next convergence rates result see [70, Theorem 3.8]. 


Theorem 4. Let Assumptions 10 and 11 hold. Moreover, let kx = kx(5, y°) be 
chosen according to the stopping rule (10) with t > 2 and let ||xo — x" |), be 
sufficiently small. Then the following estimates are valid for 6 > 0 and some positive 
constants C;- 


2(a+s) 


ke < (25 llxo— x18!) ™ (29) 
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and for-—a<r<u 


aT BT. 
lle, — xt, Scr [xo — xtra 


As usual for regularization in Hilbert scales, we are interested in obtaining 
convergence rates with respect to the norm in V = %. 


Corollary 2. Under the assumptions of Theorem 4 the following estimates hold: 


xf —xt] = o(87) if 5 <0, (30) 


lg, — x7] = O(Ilxt. —x'IIs) = 0(s**") if O<s<u. 


If in addition (28) holds, then for s > 0 the rate can be improved to 


x2, — xt] = O(Ilx?, — x'll-) = 0(s:#") if re EO <u, 


Note that (29) implies that k, is finite for 6 > 0 and hence a is a stable 
approximation of x". 

Moreover, it can be seen from (29) that the larger s the faster k. possibly grows 
if 6 — 0. As a consequence, s should be kept as small as possible to reduce the 
number of iterations and hence to reduce the numerical effort. If u is close to 0, it 
might be possible to choose a negative s. According to (30), we would still get the 
optimal rate, but, due to (29), k, would not grow so fast. Choosing a negative s 
could be interpreted as a preconditioned Landweber method (cf. [34]). 

We will now comment on the rates in Corollary 2: if only Assumption 10 (iii) is 
satisfied, i.e., if || F’(x')x|| may be estimated through the norm in 1_, only from 
above, convergence rates in ¥ can only be given if s < u, i.e., only for the case 
of undersmoothing. If s > 0, the rates will not be optimal in general. To obtain 
rates also for s > u, i.e., for the case of oversmoothing, condition (28) has to be 
additionally satisfied. From what we said on the choice of s above, the case of 
oversmoothing is not desirable. However, note that the rates for Ilxe, — x" ||9 can 
be improved if (28) holds also for 0 < s < u. Moreover, if a = a, i.e., if the usual 
equivalence condition (27) is satisfied, then we always obtain the usual optimal rates 
O(57**) (see [87]). 

For numerical computations one has to approximate the infinite-dimensional 
spaces by finite-dimensional ones. Also the operators F and F’(x)* have to be 
approximated by suitable finite-dimensional realizations. An appropriate conver- 
gence rates analysis has been carried out in [88]. This analysis also shows that a 
modification, where F’ Gi in (26) is replaced by G? (x!) similar as in (16), is 
possible. 
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Steepest Descent and Minimal Error Method 


These two methods are again of the form (11), where the coefficients wo} are chosen 
as 


5. _llsgll? s._ lly F@DIP? 
k= TD es esa. 
Fp sell? sell? 


for the steepest descent method and for the minimal error method, respectively. 

In [35] it has been shown that even for the solution of linear ill-posed problems 
the steepest descent method is only a regularization method when stopped via a 
discrepancy principle and not via an a-priori parameter choice strategy. Therefore, 
we will use (10), (14) as stopping rule. 

Again one can show the monotonicity of the errors and well-definedness of the 
steepest descent and minimal error method (see [70, Proposition 3.20]). Conver- 
gence can be shown for perturbed data (see, e.g., ['70, Theorem 3.22]). However, so 
far, convergence rates were proved only in the case of exact data (see [90]). 


Further Literature on Gradient Methods 


Iteratively Regularized Landweber Iteration 
By adding an additional penalty term to the iteration scheme of classical Landweber 
iteration, 1.e., 


ee = xe + F'(x#)*(y8 _ F(x?)) + Bx(xo — x?) with 0 < By < Bmax < }. 


one can obtain convergence rates results under weaker restrictions on the nonlin- 
earity of F (see [70, Section 3.2], [98]). The additional term is motivated by the 
iteratively regularized Gauss—Newton method, see section “Iteratively Regularized 
Gauss—Newton Method”. 


A Derivative Free Approach 

Based on an idea by Engl and Zou [37], Kiigler, in his thesis [80] (see also 
[79]), developed a modification of Landweber iteration for parameter identification 
problems where it is not needed that F is Fréchet-differentiable. 


Generalization to Banach Spaces 

A generalization of Landweber iteration to the case where VY and Y are Banach 
spaces was considered in the papers [71,99, 100], see also the book [101]. The basic 
version (in case of reflexive preimage space X’) reads as 


xh = IE GX OD + ob F'@D* IZ 08 - FD), 
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where p,g € (1,00), q* = aa) X™* is the dual of X, ie denotes a single valued 


selection from the set valued duality mapping J;¥ = 0 (2 || I"), ie a 2x" 


and ow is an appropriately chosen step size. Banach space versions of the iteratively 
regularized Landweber iteration can be found in [51, 64]. 


4 Newton Type Methods 
Newton’s method for the nonlinear operator equation (1) reads as 

F' (xp )(xt., — x2) = y® — F(x?). (31) 
Since ill-posedness of the nonlinear problem (1) is usually inherited by its lineariza- 
tion (31), regularization has to be applied in each Newton step. Formulating (31) as 
a least squares problem 


: 5 5 17.6 b\ 12 
_F _F = 
a Il_y (xj) (xz) (x — xq) I] 


and applying Tikhonov regularization leads to either the Levenberg—Marquardt 
method 


Xep1 = arg min. ly — Fx) — Fx) = xp)? + ella — all?» G2) 


where the regularization term ||x — xe ||? is updated in each Newton step, or the 
iteratively regularized Gauss-Newton method (IRGNM) 


Xie1 = arg min ly’ — FO) — P’ODG —xIP + axle — x0? G3) 


with a fixed a-priori guess x9 € . The choice of the sequence of regularization 
parameters a, and the main ideas of the convergence analysis are quite different for 
both methods as will be outlined in the following subsections. 


Levenberg—Marquardt and Inexact Newton Methods 
In the Hilbert space setting with an open set D(F’), the minimizer of the quadratic 


functional in (32) can be written as the solution of a linear system which leads to 
the formulation 


hay = XE + (F(xt)* F'(xb) + a1)! (xt) *(y' — F(XR)), (34) 


of the Levenberg—Marquardt method that can as well be motivated by a trust region 
approach. 
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Our exposition follows the seminal paper by Hanke [46], in which the first 
convergence analysis for this class of Newton type methods was given. According 
to this paper, a; should be chosen such that 


Ily5 — Fd) — F828, ox) — x2) = all — FOOD (35) 


for some q € (0, 1), where cae (a) is defined as in (34) with a, replaced by aw. This 
means that the Newton equation (31) is only solved up to a residual of magnitude 
gear (x?) || which corresponds to the concept of inexact Newton methods as 
they were first considered for well-posed problems in [28]. It can be shown (see 
[46]) that (35) has a unique solution a, provided that 


lly’ — FQ — F’ODOT — xls £ lly? — F@D| (36) 


for some y > 1 which in its turn can be guaranteed by (21). 

The key step in the convergence analysis of the Levenberg—Marquardt method is 
to show monotonicity of the error norms I|x? — x"||. To sketch this monotonicity 
proof we assume that (36) holds and therewith the parameter choice (35) is feasible. 
Using the notation K, = F’ (x?) as well as Cauchy—Schwarz inequality and the 
identity 


oy (K, Kg + 0% 1) "(9° — F(x})) = y? — FOR) - Kh, -— xd), 
we get 
Nxt — TI? = [xt = xt]? 
= 2 xhy1 — xp xb — xT) + Mea — xe? 
= ((K, Kf + oI) '(y’ — F(xd)), 
2Kx(xt — xt) + (Ky Kf + onl) 1K, Key — FO) 
— 2a ||(K, Kg + ont) "0° — FOI? 
— (KEK, + a1) | KR0* — F(x)? 
+2((K, Kg + ont) '(y' — F(x), y? — F(Qxp) — Ke (xt — xf) 


<= [xt — xt? - 2a! ly? — F(xg) — Kee. — xD - (37) 


(ily? — FO - Kea — xD - Ib’ - FQ) — Ket - xD). 
By (36) and the parameter choice (35), we have 
lly? — Fxg) — Kix? — x1 < yy? — FQ) — Ki ig — 20). 


Thus, (37) and y > 1 imply estimates (38) and (39) in the following proposition 
(see [70, Proposition 4.1]): 
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Proposition 4. Let 0 < gq < 1 < y and assume that (1) has a solution and that 
(36) holds so that a, can be defined via (35). Then, the following estimates hold: 


6 2 6 2 6 by 2 
le — xT? = Wee, — xT? = lea — xe, (38) 


IIx — x7]? = Ixty — x77 
2(y — 1) 
> Jy? — Fd) — FDO — KOI? (39) 
YOK 
2(y — l)d — qq 
> I? - FDI. (40) 
VIF? )Il 


Based on the resulting weak convergence of a subsequence of x as well as on 
quadratic summability of the (linearized) residuals, which can be easily obtained by 
summing up both sides of (39) and (40), one obtains convergence as k — oo in case 
of exact data [70, Theorem 4.2]: 


Theorem 5. Let 0 < g < 1 and assume that (1) is solvable in B,(xo), that F’ 
is uniformly bounded in B,(x"), and that the Taylor remainder of F satisfies (21) 
for some c > 0. Then the Levenberg—Marquarat method with exact data y° = y, 
\|x0 —x" || < q/c and a, determined from (35), converges to a solution of F(x) = y 
ask > ow. 


In case of noisy data, Hanke [46] proposes to stop the iteration according to 
the discrepancy principle (10) and proves convergence as 6 — 0 (see, e.g., [70, 
Theorem 4.3]): 


Theorem 6. Let the assumptions of Theorem 5 hold. Additionally let kx = 
kx (5, y°) be chosen according to the stopping rule (10) with t > 1/q. Then 
for \|xo — x"|| sufficiently small, the discrepancy principle (10) terminates the 
Levenberg—Marquardt method with a, determined from (35) after finitely many 
iterations ky, and 


kx (6, y°) = O(1+ |In6}). 


Moreover, the Levenberg—Marquardt iterates xf converge to a solution of F(x) = 
yas5 > 0. 


Convergence rates seem to be much harder to prove for the Levenberg—Marquardt 
method than for the iteratively regularized Gauss—Newton method (see section “Iter- 
atively Regularized Gauss—Newton Method”). Suboptimal rates under source con- 
ditions (8) have been proven by Rieder [94,95] under the nonlinearity assumption 
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F'(x) = Rx F(x!) and || — Ry|| <crllx — x", x € B,(xo) C D(F), 

(41) 
where cg is a positive constant. Only quite recently, Hanke [48, Theorem 2.1] 
proved the following optimal rates result: 


Theorem 7. Let a solution x* of (1) exist and let (41) as well as (8) hold with 
some 0 < yt < 1/2 and ||v|| sufficiently small. Moreover, let a, and kx be chosen 
according to (35) and (10), respectively with t > 2 and 1 > q > 1/t. Then the 
Levenberg—Marquardt iterates defined by (34) remain in B,(xo) and converge with 
the rate 


xb, — x1] = 0(857). 


Finally, we quote the rates result [70, Theorem 4.7] that is almost optimal 
and instead of the a-posteriori choices of a, and k,, presumes a geometrically 
decreasing sequence of regularization parameters, i.e., 


a, = aog* ; forsome a>O0, ge€(0,1), (42) 


and the following a-priori stopping rule 


+i +4 - € 
Mealy, <8 < mo, 2, O< Kh <ke, mrenkt+i1r, (43) 
forsome 7>0, e>0. 


Theorem 8. Let a solution x* of (1) exist and let (41) as well as (8) hold with 
some 0 < ge < 1/2 and ||v|| sufficiently small. Moreover, let a, and kx be 
chosen according to (42) and (43) with n sufficiently small, respectively. Then the 
Levenberg—Marquardt iterates defined by (34) remain in B,(xo) and converge with 
the rate 


xd, — xt = 0(6. + [Indy =), 
Moreover, 
FQ) — ll = o(sa 4 [insyi*) 
and 
kx = O(1 + |In6}). 
For the noise free case (8 = 0, n = 0) we obtain that 


Ty OW w 
IIx — x"|| = O@;), 
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and that 


| F(x) — yll = O(ap**). 


Further Literature on Inexact Newton Methods 


Hanke [47] and Rieder [94-96] have extended the Levenberg—Marquardt method by 
proposing regularization methods other than Tikhonov in the inexact solution of the 
Newton equation 


Xe = Xe + OCF (xp), y? — FO), 
with D(F'(x?), j= F(x?)), e.g., defined by the conjugate gradient method. 


Recently, Hochbruck et al. [53] proposed the application of an exponential Euler 
scheme to the Showalter differential equation 


x0) = FO)" — F@p), 
which leads to a Newton type iterative method of the form 
Xen = XE + heb (—he Fp)" F(x) Fxg)" 0" — FR), 
with 


e—1 


$(z) = ‘ 
z 


In [52] they show convergence using the discrepancy principle (10) as a stopping 
rule under condition (6), as well as optimal convergence rates under the condition 
that 


F'(x) = Ry F'(x') and || — Rx] < cr, x€B,(x')CD(F), (44) 


for some cr € (0,1), and under the source condition (8) with ~ < 1/2 for an 
appropriate choice of the pseudo time step size hy. 


Iteratively Regularized Gauss-Newton Method 


In the Hilbert space setting, the variational formulation (33) of the iteratively 
regularized Gauss—-Newton method can be equivalently written as 


xh = aE + CF ag) Fg) + oT F Op)? — Fag) + te (0 — x4)) 
(45) 
Here the sequence of regularization parameters is a-priori chosen such that 
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ak 


<r, lim a, = 0, (46) 
k->oo 


for some r > 1. 

This method was first proposed and analyzed by Bakushinskii [3], see also [5] 
and the references therein, as well as [15, 54, 66, 69, 70]. The results presented here 
and in section “Generalizations of the IRGNM” together with proofs and further 
details can be found in [70]. 

The key point in the convergence analysis of the iteratively regularized Gauss— 
Newton method is the fact that under a source condition (8) the error I|x? j= x"|| 
is up to some small additional terms equal to ap wx (4) with w;(s) defined as in the 
following lemma that is easy to prove. 


Lemma 1. Let K € L(X,Y¥), 5 € [0,1], and let {a,x} be a sequence satisfying 
a, > Oand a, — O0as k — ov. Then it holds that 


wy (s) 2= ay ||(K*K + a1)" (K*K)v < (1-5)! |v] < [I (47) 
and that 
< 
int wet = 0 O<s<l 
lvl. 5 = 


for anyv € N(A)*. 


Indeed, in the linear and noiseless case (F(x) = Kx, 6 = 0) we get from (45) 
using Kx? = y and (8) 


Xepy — XT = xp — XT + (K*K + 01)" (K*K(x1 = x4) +0 (xox 1 +-.x1—x,)) 
= —0(K*K + a1) '(K*K)“v 
To take into account noisy data and nonlinearity, we rewrite (45) as 
xb,,—xt = —ag(K*K + 041) '(K*K)4v 
= aK (KEK, + an!) '(K*K — KEK) 
(K*K + a1) |(K* K)4“v 

+ (KEK, + 1) KE(y? — Fxg) + Ke(ae—2")), (48) 

where we set Ky := F’(x’), K := F’(x‘). 

Let us consider the case that 0 < jz < 1/2 in (8) and assume that the nonlinearity 


condition (44) as well as ae € B,(x') C Bop(xo) hold. Therewith, for the Taylor 
remainder we obtain that 
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| Fxg) — Ft) — Ke(xt — x?) || < 2a || K(xg — x") I]. (49) 


The estimates [see (47)] 


NI- 


KEK, + RD WL So. KEK, FT) KE S 307 
and the identity 
K*K — K* K, = Ke(RY — Ry)K 
imply that 


lox (KEK, + an 1)7'(K* K — Ki Kx)(K*K + a 1)7'(K* K)"y|| 
-3 * * - * 
< 30,7 Ry — Ryall |K(K*K + a1) '(K* Ky 
This together with (2), (47), (48), and F (x*) = y yields the estimate (50) in 
Lemma 2 below. Inserting the identity K = R-;! K, into (48) we obtain 
XK 
Ket, =—0% K(K*K + a,1)|(K*K)“v 
— a, Ro Ki ( Ki K, + al) Ke(RQ* Ry)K 
XE Xi . 


(K*K +a1)~'(K*K)4v 
— Ri Ky (KEK, + a1) ' Kf 
k 


(F(x8) — F(xt) — Kd — xt) + y —y). 


Now the estimate (51) in Lemma 2 below follows together with (2), (44), (47), and 
(49). 

Similarly one can derive estimates (52), (53) in case of 1/2 < yu < 1 under the 
Lipschitz condition (4), by using (5) and the decomposition 


KK KR, = RK = ROS (RS EE. 


Lemma 2. Let (3), (8), (46) hold and assume that x E B,(x"). Moreover, set 
K := F'(x1), e = ad — x", and let wz (-) be defined as in (47). 


(i) If0 < w < 1/2 and (44) hold, we obtain the estimates 
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=f 
lletyill < ow, (W) + craw, (ue + 5) + 0% ?(crl|Kegl] + 48), (50) 


mek 
[Keo il] < (+ 2cr( +cr))oe” 2 w,(u + 4) 
+ (1+ cp) (2cp||Kee|| + 6). (51) 


(ii) If 1/2 < p < 1 and (4) hold, we obtain the estimates 


3 ep et 
lleta ll < aw, (W) + Lilegll Gay, *w,(H) + ||(K*K)* 2 v]]) 


1 
+ to, ?(AL Ie)? + 4), (52) 


* -i 4 * a 
Kee gill Soe (K*K)2 vl] + L*egll? Gag 7 wy) + (K* KY 24) 


2 
1 -3 * yu-4 
+ Lo |let || (a, *w, (Hu) + 5 ||(K*K)* 2 y]]) 
i = ip8 1 32 
+ (5La, ?* llegll + DGLilell- + 4). (53) 


It is readily checked that the nonlinearity condition (44) used in Lemma 2 can be 
extended to 


F'(x) = R(X, x) F’(x) + O(%,x) (54) 
|Z — R(X, x)|| < cr (55) 
|O%.x)|| < co || F’1)& - x) (56) 


for x, X € Bop(xo), where cr and cg are nonnegative constants. 
With the a-priori stopping rule 


a1 
ky > © and n= ba, >0as 6-0. (57) 
for uw = 0 and 
1 1 
noe”? <8 <nat*?, O<k <kgx, (58) 


for 0 < yj < 1 one obtains optimal convergence rates as follows (see [70, 
Theorem 4.12]): 


Theorem 9. Let (3), (8), (46) hold and let kx = ks(5) be chosen according to (57) 
for 1 = 0 and (58) for 0 < yw < I, respectively. 


(i) If0 < pw < 1/2, we assume that (54)-(56) hold and that \|xo — x*||, ||v||, 7, p, 
cr are sufficiently small. 
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(ii) If 1/2 < w <1, we assume that (4) and ||xo — x*||, ||v||, 7, p are sufficiently 
small. 


Then we obtain that 


; a Os. w=0, 
lx, — x4] = O(8747),0<pe<l, 


For the noise free case (5 = 0, n = 0) we obtain that 


ofa), O<w<1, 


ee 
Il xx Xx I oe 1, 


and that 


ubS 1 
,0< ; 

Fx) — yll = | ee) | 
O(a), 7 Su . 


With the discrepancy principle (10) as an a-posteriori stopping rule in place of 
the a-priori stopping rule (57), (58), optimal rates can be obtained under a Hoélder 
type source condition (8) with w < 5 (see [70, Theorem 4.13]): 


Theorem 10. Let (3), (8), (46), and (54)-(56) hold for some 0 < pe < 1/2, and 
let k = ks(6) be chosen according to (10) with t > 1. Moreover, we assume that 
xo — x", Ilvll, 1/7, p, and cr are sufficiently small. Then we obtain the rates 


(5 ra 0< l 
+ 0 wr’ y,US 55 
xg, — xt] = OU ie 
ov), w=h. 
In case x2 = O, and with an a-posteriori choice of a, similar to (35), the 


nonlinearity condition can be relaxed to (6), see [71]. 


Further Literature on Gauss—Newton Type Methods 
Generalizations of the IRGNM 


Already Bakushinskii in [4] proposed to replace Tikonov regularization in (45) by 
a more general method defined via functional calculus by a filter function g with 


g(a) & 7: 
Xa, = X0 + CF (xt) F (x0) F (xg)* (9? — Ft) — FOB) Go — x2). (59) 


with a; \ 0. 
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Still more general, one can replace the operator g(F’(x2)* F’(x?)) F’(x2)* by 
some regularization operator Ra(F’(x)) with 


Ry(F'(x)) & F’(x)', 


satisfying certain structural conditions so that the convergence analysis for the 
resulting Newton type method 


xf) = x0 + Ry (F (x8) (y? — F(x8) — F'x8) (0 — x8). (60) 


(see [30, 55, 59, 61, 62, 70]) applies not only to methods defined via functional 
calculus such as iterated Tikhonov regularization, Landweber iteration, and Lardy’s 
method but also to regularization by discretization. 

An additional augmentation of the analysis concerns the type of nonlinearity 
condition. Alternatively to range invariance of the adjoint of F’(x) (41), which is 
closely related to (6), one can consider range invariance of F’ (x) itself 


F'(X) = F'(x)R(%,x) and | — RO, x)|| < cr |X — x] (61) 


for x, X € Byp(xo) and some positive constant Cp. 
Incorporation of convex constraints is considered in [66, 102]. 


Generalized Source Conditions 
Convergence and optimal rates for the iteratively regularized Gauss—Newton method 
were established in [82] under a general source condition of the form 


xt—xyg = f(F' (xt) F'(x))v, ve N(F (x), 


with an index function f : [0, ||F’(x‘)||?] — [0,00] that is increasing and 
continuous with f(0) = 0. These include logarithmic source conditions (9) that 
are appropriate for severely ill-posed problems. For this purpose, it is assumed that 
conditions (54)—(56) hold and the iteration is stopped according to the discrepancy 
principle (10). 


Other A-posteriori Stopping Rules 
Bauer and Hohage in [6] carry out a convergence analysis with the Lepskii balancing 
principle, i.e., 


ka» = min {k €{0,....kmax} : [x2 — 28 || < Scam !/75 
Wm €{k + 1,...,kmac}} (62) 
(with Kmax = Kmax(6) an a-priori determined index up to which the iterates are 


well defined) in place of the discrepancy principle as an a-posteriori stopping rule. 
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Optimal convergence rates are shown for (60) with Ry, defined by Landweber 
iteration or (iterated) Tikhonov regularization under a condition similar to (54)— 
(56) if (8) with 2 < 1/2 or (9) holds, and under condition (4) if uz > 1/2 in (8). The 
advantage of this stopping rule is that saturation at 4 = 1/2 is avoided. 


Stochastic Noise Models 

In many practical applications (e.g., in weather forecast), the data noise is not only 
of deterministic nature as assumed in our exposition, but also random noise has to 
be taken into account. In [8] Bauer, Hohage, and Munk consider the noise model 


yr = F(x')+ 6n+o€ 


where 7 € Y, ||n|| < 1 describes the deterministic part of the noise with noise 
level 5, € is a normalized Hilbert space process in J (see, e.g., [13]) and o? is the 
variance of the stochastic noise. Under a Hélder source condition (8) with 4 > 1/2 
and assuming a Lipschitz condition (4), they show almost optimal convergence rates 
(i.e., with an additional factor that is logarithmic in 0) of (60) with R, defined by 
iterated Tikhonov regularization and with the Lepskii balancing principle (62) as a 
stopping rule. The setting of Hohage and Werner [57] allows for an even much more 
general setting with regard to the stochastic noise. 


Generalization to Banach Space 
Bakushinski and Kokurin in [5] consider the setting Y = ¥Y with ¥ Banach space. 
Using the Riesz—Dunford formula, they prove optimal convergence rates for the 
generalized Newton method (59) under the Lipschitz condition (4), provided a 
sufficiently strong source condition, namely (8) with 4 > 1/2 holds. 

In [71], based on the variational formulation of the iteratively regularized Gauss— 
Newton method, 


xe,, =arg min |[y? — F(xp) — F’(xg)(Qx — xf) ||? + ax |x — xoll4 
x€D(F) 


with p,q € (1,00) and Banach space norms, convergence in the general situation 
of possibly different Banach spaces VY, Y without source condition under the 
nonlinearity assumption (6) is proved. Convergence rates under variational and 
approximate source conditions generalizing (8) to the Banach space setting are 
provided in the paper [65]. 

The convergence rates results in [57] even hold for more general data misfit 
as well as regularization functionals in place of Banach space norms. Results 
on Gauss—Newton methods with other regularization than Tikhonov (similarly to 
section “Generalizations of the IRGNM”) can be found, e.g., in [58, 68]. 


Efficient Implementation 
To speed up convergence and save computational effort, it is essential to use 
preconditioning when applying an iterative regularization method Ry, in (60). 
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Egger in [32] defines preconditioners for these iterations (Landweber iteration, 
CG, or the v-methods, see, e.g., [36]) via Hilbert scales (see, e.g., [36]), which leads 
to an iterative scheme of the form 


xby, = xot+ g(L 8 F (xh)* F (xe) LF! (xb) * (9? — Ft) — F(x) (x0 — x4), 


where L is typically a differential operator and s an appropriately chosen exponent. 
It is shown in [32] that this leads to a reduction of the number of iterations to about 
the square root. 

In his thesis [81], Langer makes use of the close connection between the CG 
iteration and Lanczos’ method in order to construct a spectral preconditioner that is 
especially effective for severely ill-posed problems. 

Further strategies for saving computational effort are, e.g., multigrid [1, 42, 63, 
67,76], quasi Newton [41, 60], and adaptive discretization [72,73] methods. 


5 Nonstandard Iterative Methods 


The methods presented above were based on the standard ideas of minimizing 
a least-squares functional, namely gradient descent and Newton methods. In the 
following we shall discuss further iterative methods, either not based on descent of 
the objective functional or based on descent for a different functional than least- 
squares. 


Kaczmarz and Splitting Methods 


Kaczmarz-type methods are used as splitting algorithms for large operators. They 
are usually applied if V and F can be split into 


Y=Vi xX V2 X +++ x Yu 


and 
F= (F\, Fo,..., Fu), 


with continuous operators F; : X — J;. The corresponding least-squares problem 
is the minimization of the functional 


M 


F(x) = 35 FF) — y4114,- 


j=l 


The basic idea of a Kaczmarz-type method is to apply an iterative scheme to each 
of the least-squares terms 5 || Fj x) — ys | y, separately in substeps of the iteration. 
The three most commonly used approaches are the Landweber—Kaczmarz method 
(cf. [77]) 
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é é 6 65 6 
Xe j/M = Xk+G—pim ~ OF; 4 G—y/m) Fi Oe+G—nim) — Yi)» 
eee 


the nonlinear Kaczmarz method 


8 3 8 8 3 
Xe jim = Xeag—njm — OF; Xs jm)” Fj Oks jm) — V9) I= taeeg lt 


and the Gauss—Newton—Kaczmarz method 


6 6 6 6 —1 
Xe jim = Xk+G—yim — (Fi + G—yim)” Fj Cr+ gpm) + KD) 
5 5 5 

Fi (Xg4.G—njm)" (Fj e4G—nsm) — Y9)- 


Further Newton—Kaczmarz methods can be constructed in the same way as itera- 
tively regularized and inexact Newton methods (cf. [19]). 

The Landweber—Kaczmarz and the nonlinear Kaczmarz method can be inter- 
preted as time discretization by operator splitting for the minimizing flow 


M 
XO =-> FAO) FAO - y)), 


j=l 


with forward, respectively, backward Euler operator splitting (cf. [38, 92]). The 
nonlinear Kaczmarz method is actually a special case of the Douglas—Rachford 
splitting algorithm applied to the above least-squares problem, the iterate 54 +j/M 
can be computed as a minimizer of the Tikhonov-type functional 


5 3 
Jk j (x) = 5 F(x) Vj 5, =r x || — Xtg—nyull’. 


The convergence analysis of Kaczmarz methods is very similar to the analysis 
of the iterative methods mentioned above, if nonlinearity conditions on each single 
operator F; are posed (cf. [43, 44,77] for the Landweber—Kaczmarz, [10, 19, 83] 
for Newton—Kaczmarz, [9,27] for nonlinear Kaczmarz and further variants). The 
verification of those conditions is usually an even harder task than for the collection 
of operators F = (Fi,..., Fi), also due to the usually large nullspace of their 
linearizations. The analysis can however provide at least a good idea on the 
convergence behavior of the algorithms. A nontrivial point in Kaczmarz methods 
is an a-posteriori stopping criterion, since in general the overall residual is not 
decreasing, which rules out standard approaches such as the discrepancy principles. 
Some discussions of this issue can be found in [43], where criteria based on 
supplemented by additional skipping strategies. 

Kaczmarz methods have particular advantages in inverse problems for partial 
differential equations, when many state equations for different parameters (e.g., 
different boundary values or different sources) need to be solved. Then the operators 


460 M. Burger et al. 


F;; can be set up such that a problem for one state equation can be solved after the 
other, which is memory efficient. We mention that in this case also the Landweber 
iteration can be carried out in the same memory-efficient way, since 


M 
F'(x)* (F(x) — y°) = Do Fi (@)*(Fj (x) — v4). 
j=l 


But in most cases one observes faster convergence for the Kaczmarz-type variant, 
which is similar as comparing classical Jacobi and Gauss-Seidel methods. 

Splitting methods are frequently used for the iterative solution of problems with 
variational regularization of the form 


xé € arg min | 5 F(x) — y§|]? + aR(x)| : 


where R : X — RU {+00} is an appropriate convex regularization functional. 
It is then natural to apply operator splitting to the least-squares part and the 
regularization part, if R is not quadratic. The most important approaches are the 
Douglas—Rachford splitting (in particular for linear operators F,, cf. [31]) 


xho € argmin | FF) — yl? + sh lle x21? | 
5 = R a _ +6 2 
Ai = SE (x) + x7 IX — Xk 
and the forward—backward splitting algorithm (cf. [84]) 
Xe = Xk — OK P'(xp)" (F(R) — y°) 
1 
8 . 3 2 
Xe+1 © arg min | Ro) + ie Ix — Xe41/all 
Such algorithms are particularly popular for nonsmooth regularization (cf. [25]), 
in the case of sparsity enforcing penalties (¢'-norms) the second step in both 


algorithms can be computed explicitly via shrinkage (thresholding) formulas, such 
schemes are hence also called iterative shrinkage (cf. [26]). 


EM Algorithms 


A very popular algorithm in the case of image reconstruction with nonnegativity 
constraints is the expectation maximization (EM) method, also called Richardson— 
Lucy algorithm (cf. [12,86]). In the case of F : L'() > L'(Z) being a linear 
operator, it is given by the multiplicative fixed-point scheme 


y? 
Kg Seer 7) (63) 
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For F and F* being positivity preserving operators (such as the Radon transform 
or convolutions with positive kernels), the algorithm preserves the positivity of an 
initial value 5 if the data y® are positive, too. Positivity of data is a too strong 
restriction in the case of an additive noise model, like stochastic Gaussian models 
or bounds in squared L?-distances. It is however well suited for multiplicative 
models such as Poisson models used for imaging techniques based on counting 
emitted particles (photons or positrons). The log-likelihood functional of a Poisson 
model, respectively its asymptotic for large count rates, can also be used to derive 
a variational interpretation of the EM-algorithm. More precisely (63) is a descent 
method for the functional 


3 
J(x) := [ [tos (=) a3" + Fs| do, 


which corresponds to the Kullback—Leibler divergence (relative entropy) between 
the output Fx and the data y*®. Minimizing J over nonnegative functions leads to 


the optimality condition 
ys 
—F* (| — F*1)=0. 


With appropriate operator scaling F*1 = 1 this yields the fixed-point equation 


8 
xaxFt(Z), 
Fx 


which is the basis of the EM-algorithm: 


5 6 

3 Xk pe [ _Y 
x4, = cE F*|—_ }. 
ota (4) 


The EM-algorithm or Richardson—Lucy algorithm (cf. [103]) is a special case of the 
general EM framework by Dempster, Laird, and Rubin (cf. [29]). 

Performing an analogous analysis for a nonlinear operator F : L'(Q) > L!(Z) 
we are led to the fixed-point equation 


y3 
xF'(x)*1 = xF’(x)* (=) : 


Since it seems unrealistic to scale F’(x)* for arbitrary x it is more suitable to 


keep the term and divide by F’(x)*1. The corresponding fixed-point iteration is 
the nonlinear EM algorithm 


3 3 
Bo ME bye | 
Met = Fay (x;.) (55): 
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The convergence analysis in the nonlinear case is still widely open. Therefore, we 
only comment on the case of linear operators F here (cf. also [85,86] for details). In 
order to avoid technical difficulties the scaling condition F*1 = 1 will be assumed 
in the following. The major ingredients are reminiscent of the convergence analysis 
of the Landweber iteration, but they replace the norm distance by the Kullback— 
Leibler divergence 


KL(x.3) = | [slog = — x +3 do, 
Q Xx 


which is a nonnegative distance, but obviously not a metric. First of all, xe can be 
characterized as a minimizer of the (convex) functional 


Ay =) RL (xt 3X) 


over all nonnegative L'-functions, given 7 4,- Thus, comparing with the functional 
value at 574 4, We find that the likelihood functional J decreases during the iteration, 
more precisely 


Fa iy) = J (xf) = KL (xh41 Xp) ; 


This part of the analysis holds also in the case of exact data. The second inequality 
directly concerns the dissipation of the Kullback—Leibler divergence between the 
iterates and a solution, hence assumes the existence of xt with Fx' = y. Using 
convexity arguments one obtains 


KL(x*, xe41) + J(xe) < KL(xt, xx). 
Hence, together with the monotonicity of (J(x;)) jen 


k-1 
KL (xt, x¢) + k(x) < KL(xt, xe) + Yo (xj) < KLM, x0), 
j=0 


which implies boundedness of the Kullback—Leibler divergence (hence a weak 
compactness of x; in L!) and convergence J(x;,) — 0 analogous to the arguments 
for the Landweber iteration. 

The noisy case is less clear, apparently also due to the difficulties in defining 
a reasonable noise level for Poisson noise. An analysis defining the noise level in 
terms of the likelihood of the noisy data has been given in [93]. Further analysis in 
the case of noisy data seems to be necessary, however. This also concerns stopping 
rules for noisy data, which are usually based on the noise level. A promising 
multiscale stopping criterion based on the stochastic modeling of Poisson noise 
has been introduced and tested recently (cf. [14]). For a combination of EM with 
Kaczmarz ideas, see [45]. 
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Iterative methods are also used for Penalized EM-Reconstruction (equivalently 
Bayesian MAP estimation), i.e., for minimizing 


J(x) + aR(x) 


over nonnegative L!-functions, where aw > 0 is a regularization parameter and R is 
an appropriate regularization functional, e.g., total variation or negative entropy. 

A frequently used modification of the EM algorithm in this case is Green’s One- 
Step-Late (OSL) method (see [39, 40]). 


6 
ee Xi F* y? 
k+l" F*1 + aR'(xx) Fxi }- 


which seems efficient if the pointwise sign of R’(x,) can be controlled, e.g., for 
entropy-type regularization functionals 


R(x) = [ £@) 


with convex E : 2 — Rt and E’(0) = O. The additional effort compared 
to EM is negligible and the method converges reasonably fast to a minimizer 
of J + aR. For other important variational regularization methods, in particular 
gradient-based functionals, the OSL method is less successful, since R’(x,) does 
not necessarily have the same sign as x,, thus F*1 + a@R’(x,) can be negative or 
zero, in which case the iteration has to be stopped or some ad-hoc fixes have to 
be introduced. Another obvious disadvantage of the OSL method is the fact that 
it cannot handle nonsmooth regularizations such as total variation and £'-norms, 
which are often used to incorporate structural prior knowledge. As a more robust 
alternative, splitting methods have been introduced also in this case. In [97] a 
positivity-preserving forward—backward splitting algorithm with particular focus on 
total variation regularization has been introduced. The two-step algorithm alternates 
the classical EM-step with a weighted denoising problem 


5 Fy a 
Xet1/2 = XRF * Fx 
k 
C28 ia) 
xia € semin| [| sare . 
; k 


Convergence can be ensured with further damping, i.e., if the second half step is 
replaced by 


(x — ayx8 — (1 — wy) x3)? 
ae € arg min | ae 4 aR(x) 
~ [Joa 


3 
Xx 
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with w@, € (0, 1) sufficiently small. This algorithm is a semi-implicit approximation 
of the optimality condition 
ys 


-F' (2) +1+ep=0. Pp € OR(x), 
Fx 


where the operator-dependent first part is approximated explicitly and the reg- 
ularization part p implicitly. What seems surprising is that the constant 1 is 
approximated by x aa x, which however turns out to be crucial for preserving 
positivity. 


Bregman Iterations 


A very general way of constructing iterative methods in Banach spaces are iterations 
using so-called Bregman distances. For a convex functional R, the Bregman distance 
is defined by 


Di(X,x) = R(X) — R(x) -—(p,¥-—x), pe dR(x). 


Note that for nonsmooth R the subgradient is not single-valued, hence the distance 
depends on the choice of the specific subgradient. Bregman distances are a very 
general class of distances in general, the main properties are De (x,x) > O and 
DE Ax, x) = O. Particular cases are 


DR(&,x) = 51% — x]? for RO) = 5 [bl 


and the Kullback—Leibler divergence for R being a logarithmic entropy functional. 

If some data similarity measure H(F (x), y°) and a regularization functional R 
is given, the Bregman iteration (cf. [16,91] in its original, different context) consists 
of 


ie argmin | H(F(x), ya De Ga) 


with the dual update 
Pitt = Pk ~ Ox H(F (p41). 9°) € ARG 4A) - 


The Bregman iteration is a primal—dual method in the sense that it computes an 
update for the primal variable x as well as for the dual variable p € dOR(x). 
Consequently one also needs to specify an initial value for the subgradient po € 
OR (x8). 

Most investigations of the Bregman iteration have been carried out for H being 
a squared norm, i.e., the least-squares case discussed above 


H(F(x), y°) = 4|/F(x) —y? |’. 
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Under appropriate nonlinearity conditions a full convergence analysis can be carried 
out (cf. [2]), in general only leading to some weak convergence and convergence 
in the Bregman distance. If F is a nonlinear operator, further approximations in 
the Bregman iterations by linearization are possible leading to the Landweber-type 
method (also called linearized Bregman iteration) 


Se arg min [( F' (x?) (x — x8), F(x’) — y®) + D2 (x, xf)] 
and Levenberg—Marquardt type method 
xeq1 € argmin [5 || Fxg) + Fg) — xg) — YP? + De Ox, x6]. 


Both schemes have been analyzed in [2], see also [21—23] for the linearized Breg- 
man iteration in compressed sensing. We mention that in particular the linearized 
Bregman method does not work with an arbitrary convex regularization functional. 
In order to guarantee that the functional to be minimized in each step of the iteration 
is bounded from below so that the iterates are well defined, a quadratic part in the 
regularization term is needed. 

A discussion of Bregman iterations in the case of nonquadratic term H can be 
found in [17, 18,50] with particular focus on F being a linear operator. In this 
case also a dual Bregman iteration can be constructed, which coincides with the 
original one in the case of quadratic H, but differs in general. For this dual Bregman 
iterations also convergence rates under appropriate source conditions can be shown 
(cf. [18]), which seems out of reach for the original Bregman iteration for general 
HT. A systematic analysis of Bregman iterations in image restoration can be found 
in [20]. In [11], Bregman iterations are used to enhance generalized total variation 
and infimal convolution regularization. 


6 Conclusion 


Iterative methods offer an attractive alternative to variational regularization but are 
also closely linked to them via iterative optimization. In this chapter we aimed at 
giving a broad overview on the main classical (gradient and Newton type) as well as 
nonstandard (Kaczmarz, expectation maximization, Bregman) iterations. We put an 
emphasis on their regularizing properties for nonlinear ill-posed problems in Hilbert 
spaces and provided outlooks on further aspects such as efficient implementation, 
stochastic noise models, or formulations in Banach spaces. 
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Abstract 


In this chapter, an introduction is given into the use of level set techniques for 
inverse problems and image reconstruction. Several approaches are presented 
which have been developed and proposed in the literature since the publication of 
the original (and seminal) paper by F. Santosa in 1996 on this topic. The emphasis 
of this chapter, however, is not so much on providing an exhaustive overview of 
all ideas developed so far but on the goal of outlining the general idea of structural 
inversion by level sets, which means the reconstruction of complicated images 
with interfaces from indirectly measured data. As case studies, recent results (in 
2D) from microwave breast screening, history matching in reservoir engineering, 
and crack detection are presented in order to demonstrate the general ideas 
outlined in this chapter on practically relevant and instructive examples. Various 
references and suggestions for further research are given as well. 


1 Introduction 


Level Set Methods for Inverse Problems and Image Reconstruction 


The level set technique has been introduced for the solution of inverse problems in 
the seminal paper of Santosa [82]. Since then, it has developed significantly and 
appears to become now a standard technique for solving inverse problems with 
interfaces. However, there are still a large number of unresolved problems and 
open questions related to this method, which keeps fuelling active research on it 
worldwide. This chapter can only give a rough overview of some techniques which 
have been discussed so far in the literature. For more details which go beyond the 
material covered here, the reader is referred to the recent review articles [20, 31— 
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33,92], each of them providing a slightly different view on the topic and making 
available a rich set of additional references which the interested reader can follow 
for further consultation. 


Images and Inverse Problems 


An image, as referred to in this chapter, is a (possibly vector-valued) function which 
assigns to each point of a given domain in 2D or in 3D one or more physical 
parameter values which are characteristic for that point. An image often contains 
interfaces, across which one or more of these physical parameters change value in a 
discontinuous manner. In many applications, these interfaces coincide with physical 
interfaces between different materials or regions. These interfaces divide the domain 
Q in subdomains Q;,, k = 1,..., K of different region-specific internal parameter 
profiles. Often, due to the different physical structures of each of these regions, quite 
different mathematical models might be most appropriate for describing them in the 
given context. 

Since the image represents physical parameters, it can be tested by physical 
inspection. Here, the physical parameters typically appear in partial differential 
equations (PDEs) or in integral equations (IEs) as space-dependent coefficients, 
and various probing fields are created for measuring the response of the image to 
these inputs. Due to physical restrictions, these measurements are typically only 
possible at few discrete locations, often situated at the boundary of the domain Q, 
but sometimes also at a small number of points inside Q. If the underlying PDE is 
time dependent, then these measurements can be time-dependent functions. The 
corresponding measured data give information on the spatial distribution of the 
subdomains and on the corresponding internal model parameters. 

Sometimes the physical interpretation of the image is that of a source distribution 
rather than a parameter distribution. Then, the image itself creates the probing field 
and needs to be determined from just one set of measured data. Also combina- 
tions are possible where some components of the (vector-valued) image describe 
source distributions and other components describe parameter distributions. Initial 
conditions or boundary conditions can also often be interpreted as images in this 
spirit, which need to be determined from indirect data. It is clear that this concept of 
an image can be generalized even further, which leads to interesting mathematical 
problems and concepts. 

There is often a large variety of additional prior information available for 
determining the image, whose character depends on the given application. For 
example, it might be known or assumed that all parameter profiles inside the 
individual subregions of a domain Q are constant with known or unknown region- 
specific values. In this particular case, only the interfaces between the different 
regions, and possibly the unknown parameter values, need to be reconstructed from 
the gathered data, which, as a mathematical problem, is much better posed [37,71] 
than the task of estimating independent values at each individual pixel or voxel from 
the same data set without additional prior information on the image. However, in 
many realistic applications, the image to be found is more complicated, and even 
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the combined available information is not sufficient or adequate for completely 
and uniquely determining the underlying image. This becomes even worse due to 
typically noisy or incomplete data, or due to model inaccuracies. Then, it needs to 
be determined which information on the image is desired and which information can 
reasonably be expected from the data, taking into account the available additional 
prior information. Depending on the specific application, different viewpoints are 
typically taken which yield different strategies for obtaining images which agree 
(in an application-specific sense) with the given information. We will give some 
instructive examples further below. 

Determining an image (or a set of possible images) from the measured data, in 
the above-described sense and by taking into account the available additional prior 
information, is called here imaging or image reconstruction. In practice, images are 
often represented in a computer and thereby need to be discretized somehow. The 
most popular discretization model uses 2D pixels or 3D voxels for representing an 
image, even though alternative models are possible. Often the underlying PDE also 
needs to be discretized on some grid, which could be done by finite differences, 
finite volumes, finite elements, and other techniques. The discretization for the 
image does not necessarily need to be identical to the discretization used for solving 
the PDE, and sometimes different models are used for discretizing the image and the 
PDE. However, in these cases, some method needs to be provided to map from one 
representation to the other. In a level set representation of an image, also the level 
set functions need to be discretized for being represented in a computer. The above 
said then holds true also for the discretizations of the level set functions, which 
could either follow the same model as the PDE and/or a pixel model for the image 
or follow a different pattern. 


The Forward and the Inverse Problem 
In this chapter, it is supposed that data g are given in the form 
g= Mi, (1) 


where .@ denotes a linear measurement operator, and U are the physical states 
created by the sources q for probing the image. It is assumed that a physical model 
A(b) is given, which incorporates the (possibly vector-valued) model parameter 
b and which is able to (roughly) predict the probing physical states when being 
plugged into an appropriate numerical simulator, provided the correct sources and 
physical parameters during the measurement process were known. The forward 
operator . is defined as 


A (b,q) = MN(b)"'q. (2) 
As mentioned, A(b) is often described in a form of some partial differential 


equation (PDE) or, alternatively, an integral equation (IE), and the individual 
coefficients of the model parameter b appear at one or several places in this model 
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as space-dependent coefficients. In most applications, measurements are taken only 
at few locations of the domain, for example, at the boundary of the area of interest, 
from which the physical parameters b or the source q (or both) need to be inferred in 
the whole domain. It is said that with respect to these unknowns, the measurements 
are indirect: They are taken not at the locations where the unknowns need to be 
determined but indirectly by their overall impact on the states (modeled by the 
underlying PDE or IE) probing the image, which are measured only at few locations. 
The behavior of the states is modeled by the operator © in (2). If in & only 
b (but not q) is unknown, then the problem is an inverse parameter or inverse 
scattering problem. If in < only q (but not 5) is unknown, then the problem is 
an inverse source problem. Given measured data g, the “residual operators” & are 
correspondingly given by 


A(b,q) = L(b.q) — 8. (3) 
Given the above definitions, an image is defined here as a mapping 
a: Q—>R', 


where Q is a bounded or unbounded region in R* or in R? and n is the number 
of components of the (vector-valued) image. Each component function a,, k = 
1,...,”, represents a space-dependent physical characteristic of the domain Q 
which can be probed by physical inspection. If it appears as a coefficient of a PDE 
(or IE), it is denoted a, = bx, and if it appears as a source, it is denoted ag = qx. 
The exposition given in this chapter mainly focuses on the recovery of parameter 
distributions a, = b, and addresses several peculiarities related to those cases. 
However, the main concepts carry over without major changes to inverse source 
problems and also to some related formulations as, for example, the reconstruction 
of boundary or initial conditions of PDEs. 


2 Examples and Case Studies 


Some illustrative examples and case studies are presented in the following, which 
will be used further on in this chapter for demonstrating basic ideas and concepts on 
realistic and practical situations. 


Example 1: Microwave Breast Screening 


Figure | shows two-dimensional images from the application of microwave breast 
screening. The images of size 160 x 160 pixels have been constructed synthetically 
based on MRI images of the female breast. Three representative breast structures 
are displayed in the three images of the left column, where the value at each pixel 
of the images represents the physical parameter “static relative permittivity.” 
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Fig. 1 Three images from microwave breast screening. The three images are synthetically 
generated from MRI breast models. Left column: two-dimensional maps of the distribution of the 
static permittivity ey inside the three breast models. Right column: the corresponding histograms 
of values of €,, in each map 


A commonly accepted model for breast tissue is to roughly distinguish between 
skin, fatty tissue and fibroglandular tissue. In the images also a matching liquid is 
shown in which the breast is immersed. Inside the breast, regions can be identified 
easily which correspond to fibroglandular tissue (high static relative permittivity 
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values) and fatty tissue (low static relative permittivity values), separated by a 
more or less complicated interface. On the right column, histograms are shown 
for the distributions of static relative permittivity values inside the breast. In these 
histograms, it becomes apparent that values for fatty and fibroglandular tissue 
are clustered around two typical values, but with a broader range of distribution. 
However, a clear identification of fatty and fibroglandular tissue cannot be made 
easily for each pixel of the image based on just these values. 

Nevertheless, during a reconstruction, and from anatomical reasoning, it does 
make sense to assume a model where fatty and fibroglandular tissue occupy some 
subregions of the breast where a sharp interface exists between these subregions. 
Finding these subregions provides valuable information for the physician. Further- 
more, it might be sufficient for an overall evaluation of the situation to have a 
smoothly varying profile of tissue parameters reconstructed inside each of these 
subregions, allowing for the choice of a smoothly varying profile of static relative 
permittivity values inside each region. In the same spirit, from anatomical reasoning, 
it makes sense to assume a sharp interface (now of less complicated behavior) 
separating the skin region from the fatty/fibroglandular tissue on the one side and 
from the matching liquid on the other side. It might also be reasonable to assume 
that the skin and the matching liquid have constant static permittivity values, which 
might be known or not. If a tumor in its early stage of development is sought in this 
breast model, it will occupy an additional region of small size (and either simple 
or complicated shape and topology) and might have constant but unknown static 
relative permittivity value inside this region. 

During a reconstruction for breast screening, this set of plausible assumptions 
provides us with a complex mathematical breast model which incorporates this 
prior information and might yield an improved and more realistic image for the 
reconstructed breast (including a better estimate of the tumor characteristics) than 
a regular pixel-based inversion would be able to provide. This is so because it is 
assumed that the real breast follows roughly the complicated model constructed 
above and that this additional information is taken into account in the inversion. 

In this application, the underlying PDE is the system of time-harmonic 
Maxwell’s equations, or its 2D representative (describing so-called TM-waves), 
a Helmholtz equation. The “static relative permittivity,’ as mapped in Fig. 1, 
represents one parameter entering in the wavenumber of the Debye dispersion 
model. The electromagnetic fields are created by specifically developed microwave 
antennas surrounding the breast, and the data are gathered at different microwave 
antennas also located around the breast. For more details, see [52]. 


Example 2: History Matching in Petroleum Engineering 


Figure 2 shows a synthetically created 2D image of a hydrocarbon reservoir 
during the production process. Circles indicate injection wells, and crosses indicate 
production wells. The physical parameter displayed in the image is the permeability, 
which affects fluid flow in the reservoir. Physically, two lithofacies can be distin- 
guished in this image, namely, sandstone and shaly sandstone (further on simply 
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Fig. 2 An image from reservoir engineering. Shown is the permeability distribution of a fluid 
flow model in a reservoir which consists of a sandstone lithofacie (values in the range of 150- 
500 mDarcy) and a shaly sandstone lithofacie (values in the range of 900—1,300 mDarcy), separated 
by a sharp interface. The sandstone region shows an overall linear trend in the permeability 
distribution, whereas the shaly sandstone region does not show any clear trend 


called “shale’’). The sandstone region has permeability values roughly in the range 
150-500 mDarcy, whereas shale has permeability values more in the range 900- 
1,300 mDarcy. In petroleum engineering applications, the parameters inside a given 
lithofacie sometimes follow an overall linear trend, which is the case here inside the 
sandstone region. This information is often available from geological evaluation 
of the terrain. As a rough approximation, inside this region, the permeability 
distribution can be modeled mathematically as a smooth perturbation of a bilinear 
model. Inside the shale region, no trend is observed or expected, and therefore 
the permeability distribution is described as a smooth perturbation of a constant 
distribution (i.e., an overall smoothly varying profile). 

During a reconstruction, a possible model would be to reconstruct a reservoir 
image from production data which consists of three different quantities: (1) the 
interface between the sandstone and shale lithofacies, (2) the smooth perturbation 
of the constant profile inside the shale region, and (3) the overall trend (i.e., the 
bilinear profile) inside the sandstone region, assuming that inside this sandstone 
region the smooth perturbation is small compared to this dominant overall trend. 
In this application, the PDE is a system of equations modeling two-phase or three- 
phase fluid flow in a porous medium, of which the relative permeability is one model 
parameter. 

The “fields” (in a slightly generalized sense) are represented in this application 
by pressure values and water/oil saturation values at each point inside the reservoir 
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during production and are generated by injecting (under high pressure) water in 
the injection wells and extracting (imposing lower pressure) water and oil from the 
production wells. The data are the injection and production rates of water and oil, 
respectively, and sometimes pressure values measured at injection and production 
wells over production time. For more details, see [35]. 


Example 3: Crack Detection 


Figure 3 shows an image of a disconnected crack embedded in a homogeneous 
material. The cracks are represented in this simplified model as very thin regions of 
fixed thickness. The physical parameter represented by the image is the conductivity 
distribution in the domain. Only two values can be assumed by this conductivity, one 
inside the thin region (crack) and another one in the background. The background 
value is typically known, and the value inside the crack might either be approxi- 
mately known or it might be an unknown of the inverse problem. The same holds 
true for the thickness of the crack, which is assumed constant along the cracks, even 
though the correct thickness (the constant) might become an unknown of the inverse 
problem as well. Here insulating cracks are considered, where the conductivity is 
significantly lower than in the background. The probing fields inside the domain 
are the electrostatic potentials which are produced by applying voltages at various 
locations along the boundary of the domain, and the data are the corresponding 
currents across the boundary at discrete positions. 

This model can be considered as a special case of a binary medium where 
volumetric inclusions are embedded in a homogeneous background. However, the 
fact that these structures are very thin with fixed thickness requires some special 
treatment during the shape evolution, which will be commented on further below. 
In this application, the underlying PDE is a second-order elliptic equation modeling 
the distribution of electric potentials in the domain for a set of given applied voltage 
patterns. For more details, see [4]. 


3 Level Set Representation of Images with Interfaces 


A complex image in the above sense needs a convenient mathematical representa- 
tion in order to be dealt with in a computational and mathematical framework. In 
this section, several different approaches are listed which have been proposed in the 
literature for describing images with interfaces by a level set technique. First, the 
most basic representation is given, which only considers binary media. Afterwards, 
various representations are described which represent more complicated situations. 


The Basic Level Set Formulation for Binary Media 


In the shape inverse problem in its simplest form, the parameter distribution is 
described by 
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Fig. 3 An image from the application of crack detection. Three disconnected crack components 
are embedded in a homogeneous background medium and need to be reconstructed from 
electrostatic measurements at the region boundary. In the considered case of insulating cracks, 
these components are modeled as thin shapes of fixed thickness with a conductivity value much 
lower than the background conductivity 


bx) in D 
b(x) = ; 4 
@) bx) in Q\D 4) 
where D C Q is a subregion of & and where usually discontinuities in the 
parameters b occur at the interface 0D. In the basic level set representation for 
the shape D, a (sufficiently smooth, i.e., Lipschitz continuous) level set function 
¢ :  — R is introduced and the shape D is described by 


o(x) <0 forall xeD, (5) 
o(x) >0 forall xe Q\D. 
In other words, the parameter function b has the form 
b®(x) where (x) <0 
n= b©(x) where (x) > 0. (6) 


Certainly, a unique representation of the image is possible by just knowing those 
points where ¢(x) has a change of sign (the so-called zero level set) and additionally 
knowing the two interior profiles b“ (x) and b©)(x) inside those areas of Q where 
they are active (which are D and Q\D, respectively). Often, however, it is more 
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Q\D 


Fig. 4 The basic level set representation of a shape D. Those points of the domain where the 
describing level set function assumes negative values are “inside” the shape D described by the 
level set function ¢@, while those with positive values are “outside” it. The zero level set where 
= O represents the shape boundary 


convenient to assume that these functions are defined on larger sets which include 
the minimal sets mentioned above. In this chapter, it is assumed that all functions 
are defined on the entire domain &2, by employing any convenient extensions from 
the abovementioned sets to the rest of Q. Again, it is clear that the above extensions 
are not unique and that many possible representations can then be found for a given 
image. Which one to choose depends on details of the algorithm for constructing 
the image, on the available prior information, and possibly on other criteria (Fig. 4). 

For a sufficiently smooth level set function, the boundary of the shape D permits 
the characterization 


dD ={xeQ, (x) =0}. (7) 


This representation motivates the name zero level set for the boundary of the 
shape. In some representations listed further below, however, level set functions are 
preferred which are discontinuous across those sets where they change sign. Then, 
the boundary of the different regions can be defined alternatively as 


dD = {x € Q : forall p > 0 we can find x;, x2 € By(x) (8) 
with @(x,) > 0 and @(x2) < 0} 


where B, (xo) = {x € Q : |x — xo| < p}. 


Level Set Formulations for Multivalued and Structured Media 


As mentioned already above, in many applications the binary model described 
in section “The Basic Level Set Formulation for Binary Media” is not sufficient 
and more complex image models need to be employed. Several means have been 
discussed in the literature for generalizing the basic model to more complex 
situations, some of them being listed in the following. 
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Different Levels of a Single Smooth Level Set Function 

A straightforward generalization of the technique described in section “The Basic 
Level Set Formulation for Binary Media” consists in using, in addition to the level 
set zero, additional level sets of a given smooth (e.g., Lipschitz continuous) level set 
function in order to describe different regions of a given domain [99]. For example, 
define 


Ty ={xeQ, (x) =<} (9) 
D; => {x € Q, Ci+1 < (x) < ci}, (10) 
where c; are prespecified values with cj+; > c; fori = 0,...,i — 1 and with 
Co = +00, cj = —oo. Then, 
Q=|)D;, with DiN Dy =B fori Zi’. (11) 
i=0 
A level set representation for the image b is then given as a tupel (bo,...,b:, b) 
which satisfies 
b(x) = b;(x) for c;41 < (x) < Gj. (12) 


It is clear that certain topological restrictions are imposed on the distribution of the 
regions D; by this formulation. In particular, it favors certain nested structures. For 
more details, see [63]. 


Piecewise Constant Level Set Function 

This model describes piecewise constant multiple phases of a domain by only one 
level set function and has its origins in the application of image segmentation. A 
single level set function is used which is only allowed to take a small number of 
different values, e.g., 


o(x)=i inD;, fori =0,...,i, (13) 


Q=\|JDi. with D;ND,=9% for i¥¢i’. 
i=0 


Introducing the set of basis functions y; 


1 i 
y= — (@-—j) with a = 
Qj j=l 

pHi JAI 


(14) 


| 

= 
: 
— 


the parameter distribution b(x) is defined as 
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b=) dix. (15) 
i=1 
A level set representation for the image b is then given as a tupel (b;,...,5;, 6) 
with 
b(x) = b; where $(x) = i. (16) 


Numerical results using this model can be found, among others, in [61,65, 67, 97]. 


Vector Level Set 
In [99] multiple phases are described by using one individual level set function for 
each of these phases, i.e., 


T; = {xe€Q, (x) =0} (17) 

D; = {xEQ, gi (x) < 0}, (18) 

for sufficiently smooth level set functions ¢;, i = 0,...,i. In this model, the level 

set representation for the image b is given by a tupel (b1,..., 53, 61,...,¢7) which 
satisfies 

b(x) = bx (x) where ¢; (x) < 0. (19) 


Care needs to be taken here that different phases do not overlap, which is not 
automatically incorporated in the model. For more details on how to address this 
and other related issues, see [99]. 


Color Level Set 

An alternative way of describing different phases by more than one level set 
functions has been introduced in [95] in the framework of image segmentation and 
further investigated by [14, 25, 38, 63,92] in the framework of inverse problems. In 
this model (which also is known as the Chan—Vese model), up to 2” different phases 
can be represented by n different level set functions by distinguishing all possible 
sign combinations for these functions. For example, a level set representation 
for an image b containing up to four different phases is given by the tupel 
(b1, b2, b3, ba, 61, 62) which satisfies 


b(x) = b\(1 — H())). — H(g2)) + bo. — (61) A (2) (20) 
+b3H($1)(1 — H(¢2)) + bs (G1) A (2). 


Also here, the contrast values b,, v = 1,...,4 are allowed to be smoothly varying 
functions inside each region. The four different regions are then given by 
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Fig. 5 Color level set 
representation of multiple 
shapes. Each region is 
characterized by a different 
sign combination of the two 
describing level set functions 


D, = {x, <0 and @¢) <0} (21) 
Do = {x, ¢, <0 and dq) >0} 
D3 = {x, ¢,>0 and ¢ <0} 
Ds ={x, ¢1>0 and ¢2 > 0}. 


This yields a complete covering of the domain Q by the four regions, each point 
x € Q being part of exactly one of the four shapes D,; see Fig. 5. 


Binary Color Level Set 

An alternative technique for using more than one level set function for describing 
multiple phases, which is, in a certain sense, a combination of the piecewise constant 
level set model described in section “Piecewise Constant Level Set Function” and 
the color level set technique described in section “Color Level Set,’ has been 
proposed in [62] for the application of Mumford—Shah image segmentation. For the 
description of up to four phases by two (now piecewise constant) level set functions 
¢, and go, in this binary level set model, the two level set functions are required to 
satisfy 


¢@ €{-1,1}, or ¢?=1, ie {1,2}. (22) 
The parameter function b(x) is given by 
b(x) = 5 (bu(1 — (G2 — 1) ~ bl. — G2 + 0 (23) 
—b3(1 + 1) (G2 — 1) + bali + 2 + 1), 


and the four different regions are encoded as 
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gi =-1 and gy =-I]} (24) 
g,=-l1 and g@& =+1} 

D3 = {x, ¢=+1 and g@=-1} 
gj =+1 and & = +41}. 

A level set representation for an image b containing up to four different phases is 


given by the tupel (61, b2, b3, b4, 61, 62) which satisfies (23). For more details, we 
refer to [62]. 


Level Set Formulations for Specific Applications 


Often, for specific applications, it is convenient to develop particular modifications 
or generalizations of the above-described general approaches for describing multiple 
regions by taking into account assumptions and prior information which are very 
specific to the particular application. A few examples are given below. 


A Modification of Color Level Set for Tumor Detection 

In the application of tumor detection from microwave data for breast screening (see 
section “Example 1: Microwave Breast Screening”), the following situation needs 
to be modeled. The breast consists of four possible tissue types, namely, the skin, 
fibroglandular tissue, fatty tissue, and a possible tumor. Each of these tissue types 
might have an internal structure, which is (together with the mutual interfaces) 
one unknown of the inverse problem. In principle, the color level set description 
using two level set functions for describing four different phases would be sufficient 
for modeling this situation. However, the reconstruction algorithm as presented in 
[52] requires some flexibility with handling these four regions separately, which 
is difficult in this minimal representation of four regions. Therefore, in [52], the 
following modified version of the general representation of color level sets is 
proposed for modeling this situation. In this modified version, m different phases 
(here m = 4) are described by n = m — | level set functions in the following form 


b(x) = bi(1 — H(1)) + H(i) [bo(1 — Ho) (25) 
+H (2) {bs(1 — H(s)) + bss) | 


or 


Di ={x, 1 <0} (26) 
Dz = {x, ¢,>0 and ¢ <0} 

D3 = {x, ¢,>0 and ¢@>0 and @¢3 <0} 

D4 = {x, ¢@:>0 and ¢@2>0 and ¢3 > 0}, 
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Fig. 6 Multiple level set representation for modeling multiphase inverse problems. Left: original 
color level set technique for describing eight different phases by the different sign combinations 
of three level set functions. Center: modified color level set technique used in the model for 
early detection of breast cancer from microwave data. The possible eight regions of the color 
level set presentation are filled with four different materials in a tailor-made fashion for this 
application. Right: modified color level set technique for modeling the history matching problem 
of a water-flooding process in a petroleum reservoir. Also here the eight different regions are filled 
by a specific combination of materials characteristic for the reconstruction scheme used in this 
application. Regions with more than one subindex correspond to “characteristic regions” with 
averaged parameter values 


where bj, bo, b3, and by denote the dielectric parameters of the skin, tumorous, 
fibroglandular, and fatty tissue, respectively. In (25), ¢1, ¢2, and $3 are the three 
different level set functions indicating the regions filled with the skin, tumorous, 
and fibroglandular tissue, respectively, and the contrast values by, v = 1,...,4 
are generally allowed to be smoothly varying functions inside each region. This 
combination of m — 1 level set functions for describing m different phases has 
certain advantages with respect to the standard color level set formulation during 
the reconstruction process, as it is pointed out in [52]. On the other hand, it is 
obvious that (26) can be considered a special case of the color level set technique 
(section “Color Level Set”) where the theoretically possible 2? = 8 different values 
of the color level set description are enforced to fall into m = 4 different groups of 
characteristic values; see the central image of Fig. 6. 


A Modification of Color Level Set for Reservoir Characterization 

Another modification of the color level set technique has been used in [35] for the 
application of history matching in reservoir engineering; see section “Example 2: 
History Matching in Petroleum Engineering.” Given, as an example, n = 4 level set 
functions ¢),...,4, we define the parameter (permeability) distribution inside the 
reservoir by 


b = b\(1 — H(91)) H ($2) H (3) + b2 A (bi). — H(¢2)) A (bs) 


+b3H ($1) H($2)(1 — H(¢3)) + ba H (61) H ($2) A (3) 


bo +b 
+255 HG) = H(G2))1 — Hs) 
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b, + bs 


47248 (1 — (by) HGG2)(1 — 16s) 

472% (1 HG) — HG) Hs) 

ERE (1 — HN — HG) Hs) 27) 
where the permeability values b,, v = 1,...,4 are assumed constant inside each 


region. The four lithofacies are represented as 


D, = {x, ¢,<0O and ¢), >0 and 4; >O0} (28) 
Dz, = {x, ¢@2<0 and 3; >0 and gg; >0} 
D3; ={x, ¢@3<0 and ¢, >0O and gg) >0} 
D4, = {x, ¢,>0 and ¢, >0 and 43 >O}. 


Let in the following n = 4 be the number of lithofacies. In this model, a point 
in the reservoir corresponds to the lithofacie D;, (J = 1,...,n — 1) if ¢ has 
negative sign and all the other level set functions have positive sign. In addition, one 
lithofacie (which here is referred to as the “background” lithofacie with index / = n) 
corresponds to those points where none of the level set functions has a negative sign. 
Notice that typically this definition does not yield a complete covering of the whole 
domain Q by the four (72) lithofacies; see the right image of Fig. 6. Those regions 
inside the domain where more than one level set function are negative are recognized 
as so-called critical regions and are introduced for providing a smooth evolution 
from the initial guess to the final reconstruction. Inside these critical regions, the 
permeability assumes values which are calculated as certain averages over values 
of the neighboring noncritical regions. They are indicated in the right image of 
Fig. 6 by using multiple subindices indicating which noncritical regions contribute 
to this averaging procedure. For more details regarding this model, and numerical 
experiments for the application of reservoir characterization, see [35]. 


A Modification of the Classical Level Set Technique for Describing 
Cracks or Thin Shapes 

Cracks of finite thickness can be modeled by using two level set functions in 
a setup which amounts to a modification of the classical level set technique for 
binary media. For simplicity, assume that a crack or thin region of finite thickness 
is embedded in a homogeneous background. The classical level set technique 
described in section “The Basic Level Set Formulation for Binary Media” captures 
this situation in principle, since the crack can be interpreted as a simple shape 
(with possibly complicated topology) embedded in a homogeneous background. 
However, when it comes to shape evolution for such a crack-like structure, it is 
difficult to maintain a fixed thickness of the thin shape following the classical 
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Fig. 7 Multiple level set 
representation for modeling a 
disconnected crack. The zero 
level set of the first level set 
function defines the potential 
outline of the crack, of which 
the second level set function 
selects those parts that are 
actually occupied by the 
crack. The “ideal” crack has 
vanishing thickness, whereas 
the “real” crack modeled by 
the level set technique has a 
small finite thickness and is 
practically obtained by a 
narrow band technique 


shape evolution scheme. This is so since the classical shape evolution applies an 
individually calculated velocity field value in the normal direction at each point of 
the entire shape boundary, such that the thickness of the thin region will not be 
maintained. For crack evolution, the deformations of adjacent boundary points need 
to be coordinated in order to maintain the thickness of the crack during the entire 
shape evolution; see Fig. 9. 

A modified version of the classical level set technique has been proposed in 
[4, 77] which uses two level set functions for modeling crack propagation and 
crack reconstruction in this sense. Here, a small neighborhood (narrowband) of the 
zero level set of the first level set function defines the general outline of the crack, 
whereas the second level set function selects those parts of this band structure which 
in fact contribute to the possibly disconnected crack topology. 

In more details, given a continuously differentiable level set function ¢, and its 
zero level set 


Ty, = {x €Q : o1(x) = O}. (29) 
The normal n to Ig, is given by (61) and is pointing into the direction where @(x) > 
0. An (connected or disconnected) “ideal” (i.e., of thickness zero) crack with finite 
length completely contained inside Q is constructed by introducing a second level 


set function ¢2 which selects one or more parts from the line 'g,; see Fig. 7. This 
second level set function defines the region 


B={x€Q : g(x) < O}. (30) 
The “ideal” crack is then defined as a collection of finite subintervals of Tg, 


Slp1, $2] = Ty, 9 B. (31) 
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An ideal “insulating” crack S (of thickness zero) is then supplemented with a 
vanishing electrical current condition across this set S[¢,,¢2]. However, in the 
simplified level set model, not the ideal crack is considered, but cracks of finite 
thickness 26 > O with known conductivity 5; inside the crack and b, outside it. 
Moreover, in the insulating case, it is assumed that b; < b-. In this model, a small 
neighborhood of Ig, is introduced as 


i = {ye Q:y = x— tn (x), |t| < 6,x € Ty}, (32) 


and the above-defined “ideal crack” S is associated now with a “real crack” 
counterpart 


Ss=Th,08. (33) 
The conductivity distribution is 


b; for xe Ss 
b. otherwise 


b(x) = (34) 


in the domain 2. Certainly, the real crack can also alternatively be defined by 
Ss ={y € Q:y =x— n(x), |t| < 6, x € S}, (35) 


which would slightly change the shape of the crack at the crack tips. Here, the form 
(33) is preferred. For the numerical treatment, see [4,77]. 


4 Cost Functionals and Shape Evolution 


One important technique for creating images with interfaces satisfying certain 
criteria is shape evolution, more specifically, interface and profile evolution. The 
general goal is to start with a set of shapes and profiles as initial guess, and then let 
both, shapes and profiles, evolve due to some appropriate evolution laws in order to 
improve the initial guess with increasing artificial evolution time. The focus in the 
following will be on shape evolution, since evolution laws for interior profiles fairly 
much follow classical and well-known concepts. Evolution of a shape or an interface 
can be achieved either by defining a velocity field on the domain Q which deforms 
the boundaries of this shape or by defining evolution laws directly for the level set 
functions representing the shape. Some of these techniques will be presented next. 


General Considerations 


In many applications, images need to be evaluated for verifying their usefulness 
or merit for a given situation. This evaluation is usually based on a number of 
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criteria, among them being the ability of the image (in its correct interpretation) to 
reproduce the physically measured data (its data fitness). Other criteria include the 
consistence with any additionally available prior knowledge on the given situation or 
the closeness of the image to a set of reference images. In many cases, some form of 
merit function (often in terms of a properly defined cost functional) is defined whose 
value is intended to indicate the usefulness of the image in a given application. 
However, sometimes this decision is done based on visual inspection only. 

In general, during this evaluation process, a family of images is created and the 
merit of each of these images is assessed. Then, one or more of these images are 


selected. Let (B®, ...,09,6,..., g?) be a level set representation for the class 


of images to be considered. Then, creating this family of images can be described 
either in a continuous way by an artificial time evolution 


(00), .--b9O,OPO,....d2O), 6 € [0.tmah: 
with an artificial evolution time ¢ or in a discrete way 


(1) @® .O J) = 
(BP ate Oe sacs, ), RaLok 


with a counting index k. Usually these images are created in a sequential manner, 
using evolution laws 


© (6C),--b20,6O,.-. #200) = fl), 


with a multicomponent forcing term f(t) or update formulas 


(1) @ 4) QM \_ (1) @ .(@) ) 
(Beste pel pO) te Oe GO eee) 


with update operators F;. These evolution laws and update operators can also be 
defined on ensembles of images, which allows for statistical evaluation of each 
ensemble during the evaluation process. Any arbitrarily defined evolution law and 
set of update operators yield a family of images which can be evaluated, but typically 
those are preferred which point into a descent direction of some predefined cost 
functional. Some choices of such cost functionals will be discussed in the following. 


Cost Functionals 


In general, a cost functional can consist of various components, typically combined 
in an additive or multiplicative manner. Popular components for an image model 


b = (6%,...,b©) and ¢ = (¢, es gp) are: 
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1. Data misfit terms Y,,,,(b, d) 
2. Terms measuring closeness to a prior model inside each subdomain _J,,,,, (b, &) 
3. Terms enforcing geometric constraints on the interfaces _/,.,,,(b, o) 


In (1), the by far most popular data misfit term is the least squares misfit cost 
functional which, in general, is given as an expression of the form 


1 2 | 2 
PS nbs$) = 5 | Ab, 9)-&| = 5 |eulb.21- a] (36) 


where &(b,@) is the forward operator defined in (2) and u,,[b, @] indicates the 
simulated data at the set of probing locations M for this guess. Other choices can 
be considered as well; see, for example, [37]. 

(2) corresponds to classical regularization techniques, applied to each subdo- 
main, and is treated in many textbooks, such as [37,71]. Therefore, it is not discussed 
in this chapter. 

(3) has a long history in the shape optimization literature and in image processing 
applications. See, for example, [30,87]. A few concepts are presented in Sect. 5. 


Transformations and Velocity Flows 


The first technique discussed here is shape evolution by transformations and velocity 
flows. This concept has been inspired by applications in continuum mechanics. 
Given a (possibly bounded) domain Q C R” anda shape D C Q with boundary dD 
which, as usual, is denoted as I’. Let a smooth vector field v : R” — R"” be given 
with v-n = 0 on 0Q. A family of transformations S, proceeds by 


S:(X) = X + tv(X) (37) 


for all X € Q. In short, S; = J + tv where J stands for the identity map. This 
defines for each point (“particle”) X in the domain, a propagation law prescribed by 
the ordinary differential equation 


x(t, X) = V(t, x(t, X)), (38) 
x(0,X) = X (39) 

with the specific velocity choice 
V(t, x(t, X)) = v(X). (40) 


Physically, it corresponds to the situation where each point X of the domain travels 
with constant speed along a straight line which is defined by its initial velocity vector 
v(X). Notice that the definition (40) can with (37) also be written in a slightly more 
abstract fashion as 
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r) a 
V(t,x) = 5 Si(X) = (+5) o S~!(x). (41) 


In fact, it turns out that the ideas in the above example can be considerably 
generalized from the specific case (40) to quite arbitrary smooth vector fields V(t, x) 
describing smooth families of transformations T;(X). The generating vector field 
V(t, x) is often called “velocity field.” It can be done as follows. 

Let us given an arbitrary smooth family of transformations 7;(X) which maps 
every point X of the domain to the point x(t,X) = T;(X) at time ¢. The 
propagation of the point X over time ¢ is again described by the ordinary differential 
equations (38) and (39) where the velocity V is defined by 


V(t,x) = (57) oT (x). (42) 


Now the propagation of points is not restricted anymore to straight lines, but can 
be quite arbitrary. Vice versa, given a smooth vector field V(t, x), it gives rise to a 
family of transformations T;(X) via the differential equations (38) and (39) where 
every point X € Q is mapped by 7;(X) to the solution x(t, X) of (38), (39) at time 
t, ie., T;(X)(x) = x(t, X). For more details on this duality of transformations and 
velocity flows, see the well-known monographs [30, 87]. 

Notice that the numerical treatment of such a velocity flow in the level set 
framework leads to a Hamilton—Jacobi-type equation. Some remarks regarding this 
link are given in section “The Level Set Framework for Shape Evolution.” 


Eulerian Derivatives of Shape Functionals 


Given the framework defined in section “Transformations and Velocity Flows,” the 
goal is now to define transformations and velocity flows which point into a descent 
direction for a given cost functional. Some useful concepts on how to obtain such 
descent directions are discussed here. 

Let D = Do be a shape embedded in the domain at time t = 0. When 
the points in the said domain start moving under the propagation laws discussed 
above, the interior points of the shape, the boundary points, as well as the exterior 
points will move as well, and therefore the shape will deform. Denote the shape 
at time t by D, = T;(Do) where as before 7; is the family of transformations 
which correspond to a given velocity field V(t, x). Assume furthermore that a cost 
functional Y (x, f, D;,...) is given which depends (among others) upon the current 
shape D,;. Deformation of shape will entail change of this cost. The so-called shape 
sensitivity analysis of structural optimization aims at quantifying these changes 
in the cost due to a given velocity flow (or family of transformations) in order to 
determine suitable descent flows. 

Given a vector field V(t, x), the Eulerian derivative of the cost functional Y (D;) 
at time t = 0 in the direction V is defined as the limit 
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d£(D.V) = lim ACV ZO). (43) 


if this limit exists. The functional _%(D,) is shape differentiable (or simply 
differentiable) if the Eulerian derivative d_Y (D, V) exists for all directions V and 
furthermore the mapping V > d_Y (D, V) is linear and continuous (in appropriate 
function spaces). It is shown in [30, 87] that if _% (D) is shape differentiable, there 
exists a distribution G(D) which is concentrated (supported) on T = 0D such that 


d J (D,V) = (G(D), V(0)) . (44) 


This distribution G is the shape gradient of _Y in D, which is a vector distribution. 
More specifically, let mp denote the trace (or restriction) operator on the boundary 
I’. Then, the Hadamard-Zolésio structure theorem states that (under certain condi- 
tions) there exists a scalar distribution g such that the shape gradient G writes in the 
form G = wi(gn), where w* is the transpose of the trace operator at [ and where 
n is the normal to I’. For more details, see again [30, 87]. 


The Material Derivative Method 


A useful concept for calculating Eulerian derivatives for cost functionals is the 
so-called material and shape derivative of states u. In the application of inverse 
problems, these states u typically are the solutions of the PDEs (IEs) which model 
the probing fields and which depend one way or another on the shape D. 

Let as before V be a smooth vector field with (V,n) = 0 on 0Q, and let 7;(V) 
denote the corresponding family of transformations. Moreover, let u = u[D;] be 
a state function (of some Sobolev space) which depends on the shape D, C Q 
(denote as before Dp = D). The material derivative u[D, V] of u in the direction V 
is defined as 


a[D, Vv) = lim u[D;] © nN) - ul] (45) 


or 


aD. V(X) = tim B2IECO) = wlD]) 
t10o 


forXeQ, (46) 
4 Tt 


where the square brackets in the notation indicate the dependence of the states and 
derivatives on the shape D,; and/or on the vector field V. The material derivative 
corresponds to a Lagrangian point of view describing the evolution of the points in 
a moving coordinate system, e.g., located in the point x(t, X) = T;(X). 

The shape derivative u'[D, V] of u in the direction V in contrast corresponds to 
an Eulerian point of view observing the evolution from a fixed coordinate system, 
e.g., located in the point X. It is defined as 
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u[D;] —u[D] 


u[D, V] = fi , (47) 
or 
w[D, V(X) = lim DED MP IOD eis (48) 


t 


The shape derivative and the material derivative are closely related to each other. It 
can be shown that 


uw [D, V] = u[D, V] — V(u[D]) - V(0) (49) 
provided that these quantities exist and are well defined. Subtracting V(u[D]) - V(0) 
in (49) from the material derivative makes sure that the shape derivative actually 


becomes zero in the special case that the states u do not depend on the shape D. 
The material derivative usually does not vanish in these situations. 


Some Useful Shape Functionals 


To become more specific, some useful examples for shape functionals which have 
been applied to shape inverse problems are provided herein. 


1. Define for a given function ¢ the shape integral 


SAD) = : pOt@dx= a (x)dx (50) 


where 7p is the characteristic function for the domain D. Then the Eulerian 
derivative is given by 


4 A(D.V) = | div (Evo) dx =f ¢(V(O).n)zedT. (51) 


2. Consider the shape functional 


PAD) = if t(xdT (52) 


for a sufficiently smooth function ¢ defined on Q such that the traces on I’ exist 
and are integrable. The tangential divergence divr V of the vector field V at the 
boundary I is defined as 

divrV = (divV — (DV-n, n))|; (53) 


where DV denotes the Jacobian of V. Then, 
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d fAD.V) = | (VE, VO) + SaiveVO) AT (54) 


Be .¥ an extension of the normal vector field n on [ to a local neighborhood 
of I’. Then, the mean curvature x of T is defined as k = divp./ |p. With that, 
d £2(D, V) admits the alternative representation 


d_Jx(D,V) = i. ($ rs cx) (V(0),n) dP (55) 


3. A useful link between the shape derivative and the Eulerian derivative of the cost 
functional is 


JAD) = [ u[D]dx (56) 
which depends via the states u[D] on the shape D. Furthermore [30, 87] 
d £3(D,V) = [D.viax+ [wwiwo, n)p» dT. (57) 
4. Consider a cost functional 


$a(D) = [ c(P)dr (58) 


where ¢ is only defined at the shape boundary I’. Then we cannot use the 
characterization (49) directly, since V(¢)- V(0) is not well defined. In that case, 
the shape derivative is defined as 


CP. V] = gIP. VJ — Ve SIF) VO), (59) 
Vr being the gradient along the boundary I’ of the shape (chosen such that Vf = 


Vro + xn whenever all these quantities are well defined). Then, the Eulerian 
derivative of the cost functional _Y,(D) can be characterized as 


d $s(D,V) = [etvar+ | cvO.m)zo ar (60) 


where again k denotes the mean curvature on I’. 


The Level Set Framework for Shape Evolution 


So far, shape evolution has been discussed independently of its representation by a 
level set technique. Any of the abovementioned shape evolutions can practically be 
described by employing a level set representation of the shapes. 
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First, some convenient representations of geometric quantities in the level set 
framework are listed: 


1. The outward normal direction [74, 85] is given by 


_ Vo 
n(x) = War (61) 


2. The local curvature x(x) of dD, being the divergence of the normal field n(x), is 


K(x) = V-n(x) = V- (<5) . (62) 


3. The following relation is often useful 


5) = dap (x) 


= 63 
Von) os 


where dap is the n-dimensional Dirac delta distribution concentrated on 0D. 


Notice that the right-hand sides of (61) and (62) make sense at every point of the 
domain {2 where the level set function ¢ is sufficiently smooth, giving rise to a 
natural extension of these quantities from the boundary dD to a local neighborhood. 

Assume now that a sufficiently smooth flow field V(x, f) is given and that a shape 
D is represented by the continuously differentiable level set function @ with |V¢| # 
0 at the boundary of the shape. Then, the deformation of the shape due to the flow 
field V(x, t) in the level set framework can be obtained as follows. 

Since the velocity fields are assumed to be sufficiently smooth, a boundary point 
x remains at the boundary of dD(t) during the evolution of the shape. Let (x, t) 
be the set of level set functions describing the shape at every time of the evolution. 
Differentiating ¢(x, t) = 0 with respect to f yields 


dg dx 
a t Ves =0. (64) 


Identifying V(x, ft) to as and using (61), one arrives at 


dg 


ar 1 Vel V1) n(x, 1) = 0. (65) 


Defining the normal velocity as 


F (x,t) = V(x,t)-n(x, 1) (66) 
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the Hamilton—Jacobi-type equation for describing the evolution of the level set 
function follows as 


“é + F(x,t)-|V¢| =0. (67) 
5 Shape Evolution Driven by Geometric Constraints 


It is possible to define a shape evolution without any data misfit functional being 
involved. This type of shape evolution often occurs in applications of image 
processing or computational physics. For example, starting from an initial shape, 
the goal might be to define a shape evolution which aims at reducing the cost of 
the image with respect to one or more geometric quantities, typically encoded in 
some geometric cost functional. Based on the theory developed in Sect.4, some 
useful expressions will be derived here for calculating such descent directions. The 
then obtained geometrically driven shape evolutions can also be used for adding 
additional constraints or regularization during the shape evolution driven by data 
misfit, if desired. This is achieved practically by adding appropriate geometrical cost 
terms to the data misfit term and calculating descent directions for this combined 
cost. 


Penalizing Total Length of Boundaries 


Assume that [ = dD is a smooth submanifold in Q. The total length (or surface) 
of I’ is defined as 


Sen (D) = [ dT = [ Pode (68) 


Applying a flow by a smooth vector field V(x, t), Eq. (55) yields with € = 1 an 
expression for the corresponding change in the cost (68) which is 


d Jenr(D,V) = [ x (V(0),n) aI. (69) 


If the shape D is represented by a continuously differentiable level set function ¢, 
an alternative derivation can be given. First, using (63), write (68) in the form 


Sen (D()) = A 8($)|VO(x)| dx. (70) 
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Perturbing now ¢ > ¢ + y, formal calculation (see, e.g., [92]) yields that the cost 
functional is perturbed by 


0 fienr = / 3 Vv: Vo d 71 
(Perv) = f se@weov. 2e ax a) 
Therefore, using (62), it can be identified 
0 Aenr Vo 
= 6(¢)V-: —_ =5 72 
ae (?) vl ()K (72) 


where « is now an extension (defined, e.g., by (62)) of the local curvature to 
a small neighborhood of I’. For both representations (69) and (72), minimizing 
the cost by a gradient method leads to curvature-driven flow equations, which is 
V(0) = —xn. This curvature-dependent velocity has been widely used to regularize 
the computation of motion of fronts via the level set method [48], as well in the field 
of image processing [70], and has been introduced also recently for regularizing 
inverse problems; see, e.g., [41, 79, 82]. 

Two popular concepts related to the above shape evolution are the Mumford— 
Shah and the total variation functionals, which are frequently employed in image 
segmentation applications. This relationship is briefly described in the following. 

The popular Mumford—Shah functional for image segmentation [70] contains, 
in addition to a fidelity term inside each region of the segmented image, a term 
which encourages to shorten total curve length of the interfaces. This latter term can 
be written for piecewise constant binary media (see section “The Basic Level Set 
Formulation for Binary Media” with constant profiles in each region) as 


Ja= f |VH(¢)| dx. (73) 
Q 


Taking into account that VH(¢) = H'(¢)V@ = 6(@)|V¢ln, it is seen that 4. = 
Arent (D(¢)) as given in (70), which again yields the curvature-driven flow Eq. (72). 
For more details, see [26, 38, 95]. 

The total variation (TV) functional, on the other hand, can be written, again for 
the situation of piecewise constant binary media, as 


Fo = | Vo(@)\ dx = [be — bi| [wa@iax (74) 


Therefore, it coincides with the Mumford-Shah functional .%,, up to the factor 
|b. — b;|. Roughly it can be said that the TV functional (74) penalizes the product 
of the jump between different regions and the arc length of their interfaces, whereas 
the Mumford-Shah functional (73) penalizes only this arc length. Refer for more 
information to [25, 38]. 
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Penalizing Volume or Area of Shape 


It is again assumed that [ = dD is a smooth submanifold in Q. Define the total 
area (volume) of D as 


$oan(D) = i dx = i ote dx, (75) 


where the characteristic function yp : Q — {0,1} for a given shape D is defined 
as 


1, xeED 


0, xEQ\D. oe) 


Xod(&) = 
Applying a flow by a smooth vector field V(x, t), Eqs. (50) and (51) yield with ¢ = 1 
d Svin(D,V) = / divV(0)dx = / (V(0),n) dT. (77) 

D r 


Again, if the shape D is represented by a continuously differentiable level set 
function @, an alternative derivation can be given. First, using the Heaviside function 
H,, let us write (75) in the form 


Su(D) = . H() dx. (78) 


Perturbing as before ¢ > ¢ + y, it follows 


0 4volD _ 
( — | = [ s@w00 ax (79) 
such that 
0 fol _ 
yan 5(p). (80) 


In both formulations, a descent flow is given by a motion with constant speed in the 
negative direction of the normal n to the boundary I’, which is V(O) = —n. 


6 Shape Evolution Driven by Data Misfit 
An essential goal in the solution of inverse problems is to find an image which is able 


to reproduce the measured data in a certain sense. As far as interfaces are concerned, 
this gives rise to the need of finding descent directions for shape evolution with 
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respect to the data misfit functional. In the following, some concepts are presented 
which aim at providing these descent directions during the shape evolution. These 
concepts can be combined arbitrarily with the above-discussed concepts for shape 
evolution driven by geometric terms. 


Shape Deformation by Calculus of Variations 


Historically, the first approach for applying a level set technique for solving an 
inverse problem in [82] has used concepts from the calculus of variations for 
calculating descent directions for the data misfit functional. In many applications, 
this approach is still a very convenient way of deriving evolution laws for shapes. 
In the following, the main ideas of this approach are briefly reviewed, following 
[82]. The goal is to obtain expressions for the deformation of already existing 
shapes according to a normal velocity field defined at the boundary of these shapes. 
Topological changes are not formally included in the consideration at this stage 
(even though they occur automatically when implementing the discussed schemes 
in a level set-based numerical framework). The formal treatment of topological 
changes is a topic of active current research and will be discussed briefly in 
section “Topological Derivatives.” 


Least Squares Cost Functionals and Gradient Directions 

Typically, appropriate function spaces are needed for defining and calculating 
appropriate descent directions with respect to the data misfit cost functional. 
Without being very specific, in the following, the general notation P is used for 
denoting the space of parameters b and, if not otherwise specified, Z for denoting 
the space of measurements g. For simplicity, these function spaces are considered 
being appropriately chosen Hilbert or vector spaces. Certainly, other types of spaces 
can be used as well, which might lead to interesting variants of the described 
concepts. 

Consider now the least squares cost functional 


(2b), B(b))z » (81) 


Pee st 
Fb) = NRODIE = 5 


where (, }z denotes the canonical inner product in data space Z. Assume that #(b) 
admits the expansion 


Rb + bb) = B(b) + #(b)bb + O ((\5b||p) . (82) 
letting || ||p be the canonical norm in parameter space P, for a sufficiently small 


perturbation (variation) 6b € P. The linear operator Z'(b) (if it exists) is often 
called the Fréchet derivative of @. Plugging (82) into (81) yields the relationship 


J (b+ 8b) = f(b) + Re(#(b)*A(b), 5b), + O (ldb||3) es) 
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where the symbol Re indicates the real part of the corresponding quantity. The 
operator %'(b)* is the formal adjoint operator of %’(b) with respect to spaces Z 
and P: 


(2')*g, b) = (¢. R (yb) forallb€ P, g eZ. (84) 
The quantity 
grad, J = #'(b)*#(b) (85) 


is called the gradient direction of Y in b. 

It is assumed that the operators #'(b) and &’(b)* take into account the correct 
interface conditions at 0D, which is important when actually evaluating these 
derivatives in a “direct” or in an “adjoint” fashion. In many practical applications, 
the situation can occur that (formally) the fields need to be evaluated at interfaces 
where jumps occur. In these situations, appropriate limits can be considered. Alter- 
natively, the tools developed in section “Shape Sensitivity Analysis and the Speed 
Method” can be applied there. The existence and special form of Fréchet derivatives 
&'(b) (and the corresponding shape derivatives) for parameter distributions b with 
discontinuities along interfaces are problem specific and beyond the scope of this 
chapter. Refer to the cited literature, for example, [9, 15, 49, 53, 54, 58, 78]. In 
many practical implementations, the interface 0D is de facto replaced by a narrow 
transition zone with smoothly varying parameters, in which case the interface 
conditions disappear. 


Change of 5 Due to Shape Deformations 

Assume that every point x moves in the domain (2 a small distance y(x) and that 
the mapping x — y(x) is sufficiently smooth, such that the basic structure of the 
shape D remains preserved. Then, the points located on the boundary [T = dD will 
move to the new locations x’ = x + y(x), and the boundary I’ will be deformed into 
the new boundary I’ = dD’. Assume furthermore that the parameter distribution 
in Q has the special form (4), such that it will change as well. In the following, 
the first goal is to quantify this change in the parameter distribution b(x) due to an 
infinitesimal deformation as described above. 

Consider the inner product of 65 with a test function f 


(8b, fla = f abo Fa) dx = [ Sb(x) FD dx. (86) 
Q symdiff(D,D’) 
where the overline means “complex conjugate” and symdiff(D, D’) = (DU 


D')\(D ON D’) is the symmetric difference of the sets D and D’ (see Fig. 8). Since 
the difference in D and D’ is infinitesimal, the area integral reduces to a line integral. 
Let n(x) denote the outward normal to x. Then, the integral in (86) becomes 
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- ~._X =x + y(x) b=b, 


Fig. 8 Deformation of shapes using calculation of small variations 


(8b, fav = f (bie) 2%) yO) me FHA), (87) 


where ds(x) is the incremental arclength. Here it has been used that in the limit 
5b(x) = b; (x) — b, (x) at the boundary point x € 0D due to (4). It follows the result 


5b(x) = Wap (Gi (x) — be (x)) y(x)- n(x)) (88) 


where @gp is the n-dimensional restriction operator which restricts functions 
defined in Q to the boundary dD of the shape D (n = 2 or 3, usually). Therefore, 
65b(x) is interpreted now as a surface measure on dD. Using the n-dimensional Dirac 
delta distribution dap concentrated on the boundary 0D of the shape D, (88) can be 
written in the form 


5b(x) = (bj — be) y(x) - m(X) dap (x) (89) 


which is a distribution defined on the entire domain Q but concentrated on 0D where 
it has the same strength as the corresponding surface measure. Although, strictly 
speaking, they are different mathematical objects, they are identified in the following 
for simplicity. Compare (87) also to the classical shape or domain derivative as, for 
example, calculated in [49], focusing there on the effect of the infinitesimal change 
in the boundary of a scatterer on the far-field pattern of a scattering experiment. 


Variation of Cost Due to Velocity Field v(x) 

A popular approach for generating small displacements y(x) (as discussed in 
section “Change of b Due to Shape Deformations’’) for moving the boundary dD is 
to assign to each point in the domain a velocity field v(x) and to let the points x € Q 
move a small artificial evolution time [0, t] with constant velocity v(x). Then 


y(x) = v(x)t. (90) 


Level Set Methods for Structural Inversion and Image Reconstruction 503 


Plugging this into (89) for ¢ € [0, t], the corresponding change in the parameters 
follows as 


5b(x;t) = (bj — be) v(x) -m(x)t bap (x). (91) 


Plugging expression (91) into (83) and neglecting terms of higher than linear order 
yields 


A (b(t) — F (b0)) 


II 


Re (grad, %, 5b(x;t)), (92) 
Re (grad, J. (b; — be) v(x) -n(x)t dap (x), 


II 


or, in the limit t > 0, evaluating the Dirac delta distribution, 


dF (b) 
ot 


= re | grad, Y (b; — be)v(x) -n(x)ds(x), (93) 
t=0 aD 


where the overline means “complex conjugate” and grad, Y is defined in (85). 
Similar expressions will be derived further below using formal shape sensitivity 
analysis. (Compare, e.g., for the situation of TM-waves, the expression (97) 
calculated by using (93) with the analogous expressions (100) and (105) calculated 
by using formal shape sensitivity analysis.) 


If a velocity field v(x) can be found such that ra < 0, then it is expected 


(for continuity reasons) that this inequality holds in a sufficiently small time interval 
[0, t] and that therefore the total cost during the artificial flow will be reduced. This 
will be the general strategy in most optimization type approaches for solving the 
underlying inverse problem. See the brief discussion in section “Shape Evolution 
and Shape Optimization.” 

Notice that only the normal component of the velocity field 


F(x) = v(x) -n(x) (94) 


at the boundary dD of the shape D is of relevance for the change in the cost 
(compare the remarks made already in section “The Level Set Framework for Shape 
Evolution”). This is because tangential components of v do not contribute to shape 
deformations. In a parameterized way of thinking, they only “re-parameterize” the 
existing boundary. 


Example: Shape Variation for TM-Waves 

An instructive example is given here in order to demonstrate the above concepts 
for a practical application. Consider TM-waves in a typical imaging situation of 
subsurface imaging or of microwave breast screening as in the case study of 
section “Example 1: Microwave Breast Screening.” Assume for simplicity that the 
basic level set model of section “The Basic Level Set Formulation for Binary Media” 
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is applied here. The cost functional measuring the mismatch between calculated data 
u,,|D] corresponding to the shape D and physically measured data g is defined as 


1 
A (D) = 5 lity (P1 - €liiza) > (95) 


where the calculated measurements u,, are given as the electric field values u(x) 
at the set of receiver locations M. Using a properly defined adjoint state z(x) (see, 
e.g., [30, 34, 71] for details on adjoint states), it can be shown by straightforward 
calculation that 2’ (b)* &(b) takes the form 


(grad, Z) (x) = u(x)z(x), (96) 


where u(x) denotes the solution of the forward problem and grad, Y is defined as 
in (85). Therefore, it follows that 


df (b) 
ot 


= Be i u(x)z(x)(b; — be)v(x) -m(x)ds(x), (97) 
aD 


t=0 


where it is used that the real part of a complex number and its complex conjugate 
are identical. Similar expressions based on adjoint field calculations can be derived 
for a large variety of applications; see, for example, [34, 39, 45,71, 84, 89, 90]. 


Example: Evolution of Thin Shapes (Cracks) 

Another application of this technique has been presented in [4,77] for finding cracks 
in a homogeneous material from boundary data; see section “Example 3: Crack 
Detection.” The evolution of cracks as defined in section “A Modification of the 
Classical Level Set Technique for Describing Cracks or Thin Shapes” requires the 
simultaneous consideration of two level set functions. Evolution of the first level 
set function amounts to displacement of the thin region (crack) in the transversal 
direction, whereas evolution of the second level set function describes the process 
of crack growth or shrinkage in the longitudinal direction, which comes with the 
option of crack splitting and merging. Descent directions for both level set functions 
can be calculated by the following arguments presented above. It needs to be taken 
into account, however, that, due to the specific construction of a crack with finite 
thickness, deformation of the zero level set of the first level set function is associated 
with a displacement of the crack boundary at two adjacent locations, which both 
contribute to a small variation in the cost. See Fig. 9. 

Assume that a small displacement is applied to the zero level set of the first level 
set function defining S' in the notation of section “A Modification of the Classical 
Level Set Technique for Describing Cracks or Thin Shapes.” This is reflected by two 
contributions to the least squares data misfit cost, one from the deformation of S~ 
in Fig. 9 and the other one from the deformation of S+ in Fig. 9. It follows that a 
descent velocity is now given as v(x) = Fy, (x)n(x) with 


Fo, (x) = —(b; — be) [grad, %|5+ — grad, Z|s—] on S, (98) 
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Fig. 9 Deformation of a thin 
shape (crack) using calculus 
of small variations 


with grad, % being defined in (85). In (98), for each x € Ty,, two adjacent points 
of grad, %|5+ and grad, %|s- contribute to the value of F,,(x) which can be 
found in the normal direction to Tg, in x. 

In a similar way, a descent direction with respect to the second level set function 
g2 can be obtained. Its detailed derivation depends slightly on the way how the 
crack tips are constructed in section “A Modification of the Classical Level Set 
Technique for Describing Cracks or Thin Shapes,” where two alternative choices 
are given. Overall, a descent velocity can be calculated following classical rules for 
those points x of 0B (i.e., for those points of the zero level set of 62) which satisfy 
x € BN ey or, alternatively, x € 0B  Ty,. Then, the obtained velocity field 
needs to be extended first to the remaining parts of 0B and then to the rest of Q. 
Notice that the specific form of grad, % might be slightly different here from the 
one given in (96) due to the slightly different PDE which might be involved here 
(depending on the application). For more details, refer to [4, 77]. 


Shape Sensitivity Analysis and the Speed Method 


In this section, an alternative technique is presented for formally defining shape 
derivatives and modeling shape deformations driven by cost functionals. This 
theory, called shape sensitivity analysis, is quite general and powerful, such that it is 
used heavily in various applications. Only very few concepts of it can be mentioned 
here which are employed when calculating descent directions with respect to data 
misfit cost functionals. 

The tools, as presented here, have been used and advanced significantly during 
the last twenty years in the framework of optimal shape design [30, 87]. Having this 
powerful theory readily available, it is therefore quite natural that these methods 
have been applied very early already to the applications of shape-based inverse 
problems with level sets. The theory of this section is again mainly concentrated 
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upon modeling in a formally accurate way the deformation of already existing 
shapes. It does not incorporate topological changes. These will be discussed briefly 
in section “Topological Derivatives.” 


Example: Shape Sensitivity Analysis for TM-Waves 

Again the situation of inverse scattering by TM-waves is considered here. The below 
discussion closely follows the results presented in [64]. The main tools used here 
are the material and shape derivative defined in section “The Material Derivative 
Method.” 

The cost functional measuring the mismatch between calculated data u,,[D] 
corresponding to the shape D and physically measured data g is defined by (95). 
When perturbing the shape by a velocity field V(t, x), the electric field at the (fixed) 
probing line changes according to u > u+ u’, where w’ is the shape derivative 
defined in section “The Material Derivative Method.” Plugging this into (95) and 
neglecting terms of higher than linear order, it is verified that 


d J(D,V) = Re iF u' (x) (uy, — &)(x)dx. (99) 


Now, the shape derivative wu’ can be calculated by first computing the material 
derivative (also defined in section “The Material Derivative Method’) and then using 
one of the relationships between the material derivative and the shape derivative (see 
sections “The Material Derivative Method” and “Some Useful Shape Functionals”’). 
Using also here an adjoint state z, the Eulerian derivative can be characterized and 
calculated as 


d_ £(D,V) = Re / (Bint — Dext)u(x)z(x)V (0, x) nd. (100) 
T 


Notice that this is exactly the same result as we arrived at in (97). For more details, 
refer to [64]. 


Shape Derivatives by a Min—Max Principle 

In order to avoid the explicit calculation of material and shape derivatives of the 
states with respect to the flow fields, an alternative technique can be used as reported 
in [29, 30, 78]. It is based on a reformulation of the derivative of a shape functional 
J (D) with respect to time as the partial derivative of a saddle point (or a “min— 
max”) of a suitably defined Lagrangian. In the following, the basic features of this 
approach will be outlined, focusing in particular on the shape derivative for TM- 
waves. 

Let again the cost functional _Y (D(t)) be defined as in (95) by 


1 
J (Dt) = 5 Il¢w DO] — Sllr2cay - (101) 
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The goal is to write Y (D(t)) in the form 


A (D(t)) = minmax L(t, u, z) (102) 


for some suitably defined Lagrangian &(t, u,z). Here and in the following, the 
complex nature of the forward fields u and the adjoint fields z is (partly) neglected 
in order to simplify notation (more rigorous expressions can be found in [78]). The 
Lagrangian /(t, u, z) takes the form 


2 
(103) 


L(t,u,zZ) = =f fre ) Gir ( (x, x) u u (x’) dx’ — g(x) 


+ re [ (u (x) — u'"® (x) — [ne ') Gap (x, x’) u () dx’) 2(x) dx. 


Next, it can be shown that this Lagrangian has a unique saddle point denoted by 
(u*,z*), which is characterized by an optimality condition with respect to u and 
z. In fact, the uniqueness follows from the well-posedness and uniqueness of the 
solutions of the direct and adjoint state equations; see [31,78]. The key observation 
is now that 


d J 0 . a ra 
a (minmax (7.1.2) = ape eo ,Z) (104) 


which says that the time derivative of the original cost functional can be replaced 
by the partial time derivative of a saddle point. Following these ideas, the result is 
derived 


cf = Re i (Bint — Dext)u(x)z(x) V(0) -n dT (105) 
r 

which holds for TM-waves and which is identical to the previously derived 

expressions (97) and (100). 

Similar expressions (now involving expressions of the form Vu(x)Vz(x) rather 
than u(x)z(x) at the interfaces) can be derived also for so-called TE-waves; see [78]. 
The above outlined min-max approach is in fact wide ranging and can be extended 
to 3D vector scattering, geometrical regularizations, simultaneous searches of shape 
and contrast, etc. It de facto applies as soon as one has well-posedness of the direct 
and adjoint problems. For more details, refer to [29, 30, 78, 79]. 


Formal Shape Evolution Using the Heaviside Function 


A third possibility for describing and modeling shape deformations driven by data 
misfit (in addition to using calculus of variation as in section “Shape Deformation by 
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Calculus of Variations” or shape sensitivity analysis as in section “Shape Sensitivity 
Analysis and the Speed Method”) is the use of the characteristic function and formal 
(basic) distribution theory. In contrast to the previous two techniques which first 
calculate velocity fields in the normal direction to the interfaces and then move the 
interfaces accordingly using a level set technique (or any other computational front 
propagation technique), typically leading to a Hamilton—Jacobi-type formalism 
(compare the remarks in section “The Level Set Framework for Shape Evolution’), 
the method presented in the following does not explicitly use the concept of 
velocity vector fields, but instead tries to design evolution laws directly for the 
describing level set functions (thereby not necessarily leading to Hamilton—Jacobi- 
type evolution laws). 

Notice that many of the level set formulations presented in Sect.3 give rise to 
similar concepts as discussed in the following. On the other hand, typically also 
the concepts discussed in sections “Shape Deformation by Calculus of Variations” 
and “Shape Sensitivity Analysis and the Speed Method” can be translated, once 
suitable velocity fields have been determined, into level set evolutions using the 
various representations of Sect.3. Details on how these evolution laws can be 
established can be found in the literature cited in Sect. 3. 

The formalism discussed in the following is in fact very flexible and quite easy 
to handle if standard rules for calculations with distributions are taken into account. 
Moreover, it often leads to very robust and powerful reconstruction algorithms. 
Certainly, it also has some limitations: in the form presented here, it is mainly 
applicable for “penetrable objects” with finite jumps in the coefficients between 
different regions. This means that it does not generally handle inverse scattering 
from impenetrable obstacles if very specific and maybe quite complicated boundary 
conditions need to be taken into account at the scatterer surfaces. For those 
applications, the theory based on shape sensitivity analysis is more appropriate. 
Nevertheless, since many inverse scattering problems can be described in the form 
presented in the following, possibly incorporating some finite jump conditions of the 
forward and adjoint fields or their normal components across the interfaces (which 
can be handled by slightly “smearing out” these interfaces over a small transition 
zone), this theory based on formal distribution theory provides an interesting 
alternative when deriving level set-based shape evolution equations for solving 
inverse scattering problems. 

The main idea of this technique is demonstrated by giving two examples 
related to the test cases from sections “Example 1: Microwave Breast Screening” 
and “Example 2: History Matching in Petroleum Engineering.” 


Example: Breast Screening-Smoothly Varying Internal Profiles 

In the example of breast screening as discussed in section “Example 1: Microwave 
Breast Screening,” three level set functions and four different interior parameter 
profiles need to be reconstructed from the given data simultaneously. Some of the 
interior profiles are assumed to be constant, whereas others are smoothly varying. 
In the following, the theory is developed under the assumption that all interior 
parameter profiles are smoothly varying. The case of constant parameter profiles 
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in some region is captured in the next section where a similar case is discussed for 
the application of reservoir engineering. 

Let Z& (b(¢1, $2, $3, b1, bz, b3, b4)) denote the difference between measured data 
and data corresponding to the latest best guess (41, $2, 63, D1, bz, b3, b4) of level 
set functions and interior profiles in the image model discussed in section “A 
Modification of Color Level Set for Tumor Detection.” Then, the least squares data 
misfit for tumor detection is given by 


F (B(G1. 283-1. babs. bs)) = 5 ROU. 2.3. br, babs, ba)IP. 
(106) 
Introducing an artificial evolution time ¢ for the above specified unknowns of the 
inverse problem, the goal is to find evolution laws 


doy 

* = f(x), v=1,...,3, (107) 
2 (i), PH 1224 (108) 
ai = Sy (Xl), V = Lye ee g ts 


such that the cost _Y decreases with increasing evolution time. With level set 
functions and interior profiles evolving, also the cost will change, Y = Y(t), 
such that formally its time derivative can be calculated by using the chain rule 


df  dJS| cr db db, | db db, 
dt db bp a 


Here, Re indicates to take the real part of the following complex quantity and 
(_, )p denotes a suitable inner product in parameter space P, and grad, / is 
defined in (85). It is verified easily that in the situation of section “A Modification 
of Color Level Set for Tumor Detection” 


spy = Bb) ( — Br + ball — HG) (110) 
+ H(){ba(1 — H(s)) + bsH(s)}). 

ie = H($1)5($2) [-b2 + b3, 1 — H(¢3)) + b4H(43)] , (11) 

0b 

on > A ($1) H($2)5(3) {—b3 + ba}, (112) 
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and 

ab 

ab) = 1— H(¢1), (113) 

ab 

a A(¢,)U — H(¢2)), (114) 
2 
b 

a = H($\)H(¢2)0 — H(¢3)), (115) 
3 

ab 

7 ($1) H(¢2) (93). (116) 
A 


Descent directions are therefore given by 


f(t) =—C,(t) Re jerad, 7 5 ,v=1,...,3, (117) 
: ab 
g(t) =—C,(t) Re jerad, 7 5 | ,v=l,...,4, (118) 


with some appropriately chosen positive-valued speed factors C,(t) and Cy (t). An 
efficient way to compute grad, Y is again to use the adjoint formulation; see (96) 
and for more details [34, 52,71]. 

Notice that is might be convenient to approximate the Dirac delta on the right- 
hand side of (110)-(112) in the formulation of the level set evolution by either a 
norrowband scheme or by a positive constant which allows for topological changes 
in the entire computational domain driven by the least squares data misfit. For 
more details, see the brief discussion in section “Shape Evolution and Shape 
Optimization” and the slightly more detailed discussions held in [31,52]. Following 
the latter scheme, one possible numerical discretization of the expressions (107) and 
(108) in time t = ¢),n = 0,1,2,..., then yields the update rules 


ab |” 
grt) =o —81™C, (t™) Re Eeard PS Thay de O19) 


(n) 
porn = b”) _ 6t™C, (¢) Re [era 7 5 ,v= 1, aatie 4. (120) 


Example: Reservoir Characterization-Parameterized Internal Profiles 

In the example of history matching in reservoir engineering as discussed in 
section “Example 2: History Matching in Petroleum Engineering,” one level set 
function and two interior parameter profiles need to be reconstructed from the 
given data, where one interior parameter profile is assumed to be smoothly varying, 
and the other one is assumed to overall follow a bilinear pattern. The case of 
smoothly varying interior parameter profiles is completely analogous to the situation 
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discussed in the previous section for microwave breast screening, such that it is 
not considered here. In the following, the situation is treated where both interior 
profiles follow a parameterized model with a certain set of given basis functions. 
In the history matching application as well as in the microwave breast screening 
application, the mixed cases of partly parameterized (e.g., with a constant or a 
bilinear profile) and partly smooth profiles are straightforward to implement as 
combinations of these two general approaches. 

In this approach, it is assumed that the two internal profiles can be written in the 
parameterized form 


Ni Ne 
bi(x) = So ajaj(x), be(x) = D> Bebe (x), (121) 


j=l k=1 


where a; and by are the selected basis functions for each of the two domains D 
and {2 — D, respectively. See the model discussed in section “A Modification of 
Color Level Set for Reservoir Characterization.” In the inverse problem, the level 
set function ¢ and the weights a; and 6; need to be estimated with the goal 
to reproduce the measured data in some sense. In order to obtain an (artificial) 
evolution of the unknown quantities ¢, a;, and Bx, the following three general 
evolution equations for the level set function and for the weight parameters are 
formulated 


dp _ 

ae S(x.t,¢, 4), (122) 
daj _ dpe _ 
i gji(t.¢,Z), “wa hy (t,, &). (123) 


In the same way as before, the goal is to define the unknown terms f, g;, and hy 
such that the mismatch in the production data decreases during the evolution. For 
this purpose, we reformulate the cost functional now as 


1 
J (b(o, 4, Be) = sIAOO.4;, Be), (124) 


where a; denotes the weight parameters for region D and f, denotes the weight 
parameters for region Q — D. Formal differentiation of this cost functional with 
respect to the artificial time variable ¢ yields, in a similar way as before, the descent 
directions [35] 


Ssv(x) = —Cixwa()(be — bi )grad, J, (125) 


esc) [ a;(1— H(¢))grad, fax, (126) 


tsa) = =CGd) i by H(o)grad, fax, (127) 
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where C;, C(a@;), and C(f,) are again positive-valued speed factors which are used 
for steering the speed of evolution for each of the unknowns @¢, a;, and Bx individu- 
ally. The narrowband function yyg(@) is introduced for computational convenience 
and can be omitted if desired. For details on this narrowband formulation, see the 
brief discussion held in section “Shape Evolution and Shape Optimization.” 


7 Regularization Techniques for Shape Evolution Driven by 
Data Misfit 


Regularization of shape evolution can be achieved by additional additive or multi- 
plicative terms in the cost functional which control geometric terms, as discussed 
in Sect. 5. Alternatively, some form of regularization can be obtained by restricting 
the velocity fields, level set updates, or level set functions to certain classes, often 
without the need to introduce additional terms into the cost functional. Some of 
these techniques are presented in the following. 


Regularization by Smoothed Level Set Updates 


In the binary case (see section “The Basic Level Set Formulation for Binary 
Media’), a properly chosen level set function @ uniquely specifies a shape D[@]. 
This can be described by a nonlinear operator IT mapping level set functions to 
parameter distributions 


bi(x), (x) <9, 


TI(p)(x) = be(x), (x) > 0. (128) 
We obviously have the equivalent characterization 
T1(p)(x) = bi(&)Xd(&) + be(x)(1 — x(x) (129) 


where 7 p is the characteristic function of the shape D. The “level set-based residual 
operator” 7 (#) follows as 


T(o) = AM). (130) 


Formal differentiation by the chain rule yields 


T'(o) = #(M(¢)) 0). (131) 


The (formal) gradient direction of the least square cost functional 


> 1 
4 (b) = 5 IFO (132) 
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is then given by 
grad 2(¢) = 7'(¢)* 70), (133) 


where 7’(¢)* is the Lz-adjoint of 7’(¢). Moreover, formally it is calculated by 
standard differentiation rules that 


TI'(¢) = (bj — be)5(4). (134) 


Notice that, strictly speaking, the right-hand side of (131) is not an L2-function due 
to the Delta distribution which is seen in (134). Nevertheless, in order to obtain 
practically useful expressions in a straightforward way, it is convenient to proceed 
with the formal considerations and, whenever necessary, to approximate the Dirac 
delta distribution 5(@) by a suitable L2-function; see the brief discussion on this 
topic held in section “Shape Evolution and Shape Optimization.” For example, the 
narrowband function 7 q(x) as defined in (151) can be used for that purpose. Then, 


Toy = (Gy A401)". (135) 


Assuming now that @ € W,(Q) with 
dg 
W,(Q2) = 66:6 € Lo(Q), Ved € La(Q), a 0 at OQ? , (136) 
v 


the adjoint operator .7’(¢)* needs to be replaced by a new adjoint operator .7’(@)° 
which maps back from the data space into this Sobolev space W;(Q2). Using the 
weighted inner product 


(v, w) (2) = av, W)1.(2) + B(Vv, Vw) 1,9) (137) 


with w > 1 and 6 > 0 being carefully chosen regularization parameters, it follows 


T'(p)? = (al — BA)" F'(6)*. (138) 


The positive definite operator (wJ — BA)! has the effect of mapping the L 
gradient .7’(¢)* 7() from L2(Q) towards the smoother Sobolev space W,(Q). 
In fact, different choices of the weighting parameters a and f visually have 
the effect of “smearing out” the unregularized updates to a different degree. In 
particular, high-frequency oscillations or discontinuities of the updates for the level 
set function are removed, which yields shapes with more regular boundaries. Notice 
that for @ € W,(), the trace $|, (which is the zero level set) is only within 
the intermediate Sobolev space Wj/2(I") due to the trace theorem. Therefore, the 
“degree of smoothness” of the reconstructed shape boundaries I" lies somewhere in 
between L2(T) and W,(T). 
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Sometimes it is difficult or inconvenient to apply the mapping (138) to the 
calculated updates 7’(¢)* 7(p). Then, an approximate version can be applied 
instead which is derived next. Denote f. = 7'()° Z(¢) and f, = 7'(¢)* Z(@). 


f, can formally be interpreted as the minimizer of the cost functional 


» a—1 B 1 
Bf) = SW li + SIV IME, + SIP — fl (139) 
In particular, the minimization process of (139) can be attempted by applying a 
gradient method instead of explicitly applying (aJ — BA)! to f,. The gradient 
flow of (139) yields a modified heat (or diffusion) equation of the form 


vy, - BAv= f,-—av for t¢ € [0,Tt] (140) 
v(0) = f,,, 


with time-dependent heating term f, — av, where ) = v(t) evolves towards the 
minimizer f, of (139) for t — oo. Practically, it turns out that a satisfactory 
regularization effect is achieved if instead of (140) the simplified heat equation is 
solved for a few time steps only: 


vy, -BAv=0 for te [0,T] (141) 
v(0) = fa, 


for t small, using > = v(t) instead of f, as update. For more details, see [44]. 

The above-described regularization schemes only operate on the updates (or 
forcing terms f in a time-dependent setting) but not on the level set function itself. 
In particular, in the case that a satisfactory solution of the shape reconstruction 
problem has already been achieved such that the data residuals become zero, the 
evolution will stop (which sometimes is desirable). In the following subsection, 
we will mention some alternative regularization methods where the evolution in 
the above-described situation would continue until an extended cost functional 
combining data misfit with additional geometric terms or with additional constraints 
on the final level set functions is minimized. 


Regularization by Explicitly Penalizing Rough Level Set Functions 


Instead of smoothing the updates to the level set functions, additional terms can be 
added to the data misfit cost functional which have the effect of penalizing certain 
characteristics of the level set function. For example, a Tikhonov—Philips term for 
the level set function can be added to (81), which will yield the minimization 
problem 


1 
min J (4) = 5 |2b@))z + P). (142) 
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where || ||z denotes the canonical norm in the data space Z and where p denotes 
some additional regularization term, typically involving the norm or semi-norm 
in the space of level set functions, for example, p(¢) = Volz, A discussion 
of different choices for p(#) is provided in [93]. Alternative functionals could be 
applied to the level set function ¢, as, for example, Mumford—Shah, total variation, 
etc., which would allow for jumps in the representing level set functions. 


Regularization by Smooth Velocity Fields 


In the previous two subsections, regularization tools have been discussed, which are 
directly linked to the level set formulation of shape evolution. In section “Regular- 
ization by Smoothed Level Set Updates,’ smoothing operators have been applied 
to the updates of the level set functions (or forcing terms) which are considered 
as being defined on the whole domain Q. The additional terms discussed in 
section “Regularization by Explicitly Penalizing Rough Level Set Functions,” on 
the other hand, will yield additional evolutions terms which typically have to be 
applied directly to the describing level set functions during the shape evolution. 

An alternative concept of regularizing shape evolution, which does not directly 
refer to an underlying level set representation of the shapes, consists in choosing 
function spaces for the normal velocity fields which drive the shape evolution. These 
velocity fields are, as such, only defined on the zero level set, i.e., on the boundaries 
of the given shapes (unless extension velocities are defined for a certain reason). For 
example, the velocity field could be taken as an element of a Sobolev space W, (I) 
equipped with the inner product 


ova 
(vw) nar) = / (Fo +m) ds, (143) 
T 


where ds is the surface increment at the boundary. This leads to a postprocessing 
operator applied to the gradient directions which are restricted to the boundary I’. 
The action of this postprocessing operator can be interpreted as mapping the given 
velocity field from L2(I) towards the smoother subspace W,(I°), much as it was 
described in section “Regularization by Smoothed Level Set Updates” for the spaces 
L2(&2) and W,(&2). For the above given norm (143), this is modeled by a Laplace— 
Beltrami operator 


ay 


— aa +v= Sale. (144) 


Weighted versions of (143) and (144), with parameters a and 6 as in (137), can 
be defined as well. These operators have the effect of smoothing the velocity 
fields along the boundary I and therefore lead to regularized level set evolutions 
if suitable extension velocities are chosen. Alternatively, diffusion processes along 
the boundary can be employed for achieving a similar effect of smoothing velocity 
fields. For a more detailed description of various norms and the corresponding 
surface flows, see [17,50, 87]. 
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Simple Shapes and Parameterized Velocities 


An even stronger way of regularizing shape evolution is to restrict the describing 
level set functions or the driving velocities to be members of finite-dimensional 
function spaces spanned by certain sets of basis functions. As basis functions, 
for example, polynomials, sinusoidal or exponential functions, or any other set 
of linearly independent functions tailored to the specific inverse problem, can be 
used. Closely related to this approach is also the strategy of restricting the shapes 
(and thereby the shape evolution) to a small set of geometric objects, as, for 
example, ellipsoids. See the discussion in [31] where evolution laws for a small 
sample of basic shapes are derived. In a related manner, [11] considers a multiscale 
multiregion level set technique which adaptively adjusts the support and number 
of basis functions for the level set representation during the shape evolution. Also 
related to this approach is the projection mapping strategy for shape velocities as 
proposed in [12]. 


8 Miscellaneous On-Shape Evolution 


Shape Evolution and Shape Optimization 


Shape evolution and shape optimization are closely related. Assume given any 
velocity function F(x) = v(x)-n(x) pointing into a descent direction of the cost 


J , such that 2) <0. Then, the cost will decrease in the artificial time 


t= 
evolution during a sufficiently small time interval [0, t]. On the practical level, the 
corresponding Hamilton—Jacobi-type evolution equation for the representing level 
set function to be solved during the time interval [0, t] reads 


dg 


_ + F|V¢| = 14 
oo |Vo| = 0, (145) 


where the variables (x, f) have been dropped in the notation. Using, for example, a 
straighforward time-discretization scheme with finite differences yields 


p(t) - (0) + F\V¢| = 0. (146) 


Interpreting @"t) = (zr) and 6”) = (0) yields the iteration 
gary = ge” 4 15p”, gO = do, (147) 


where t plays the role of the step size (which might be determined by a line-search 
strategy) and where 


56 = FIVG™| (148) 


for x € 0D. 
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In the level set optimization approach, on the other hand, updates v for a level 
set function @ — ¢@ + v are sought which reduce a given cost. Take, for example, 
the situation of the basic level set formulation described in section “The Basic Level 
Set Formulation for Binary Media.’ Analogously to section “Least Squares Cost 
Functionals and Gradient Directions,” a small perturbation v then has the effect on 
the cost 


df df db _ db 
o> Ib i Re(eraa, 7, 5°), (149) 


= Re (grad, JZ, (be (x) — b; (x))6(¢)v), , 


with grad, Y% defined in (85). Apart from the term 4(@), this yields similar 
expressions for the discrete updates as (147) if choosing F|V@™| = — Re(b,(x)— 
b;(x))grad, 7 . 

In fact, the term 6(@) is the one which causes the biggest conceptual problem 
when interpreting the above scheme in an optimization framework. Notice that, 
strictly speaking, the mapping from the level set function to the data (or to the 
corresponding least square cost) is not differentiable in standard (e.g., L2) function 
spaces. This is indicated by the appearance of this Dirac delta distribution 5(@) in 
(149), which is not an L> function. 

There are several ways to circumvent these difficulties, mainly aiming at 
replacing this troublesome Delta distribution by a better behaved approximation of 
it. First, in the narrowband approach, the Dirac delta is replaced by a narrow band 
function 7g 4¢(x) which yields 


F,|Vo|(x) = — Re ((be — bi) xg.a(x) grad, Y) forall xeQ. (150) 


Here, 74,¢(x) is an arbitrary positive-valued approximation to 5(@) where the 
subscript d indicates the degree of approximation. For example, it can be chosen 
as 


1, there exists x9 € Q with |x — xo| < d and #(xo) = 0 


151 
0, otherwise ee 


Xo.d(x) = 


which has the form of a “narrowband” function. Other approximations with 
certain additional properties (e.g., on smoothness) are possible as well. This search 
direction obviously also provides a descent flow for _¥%. In fact, the term |Vo”| 
can also be neglected in (150), without losing the descent property of the resulting 
flow, since formally it can be assumed that |V¢| > 0 (repeated recalculation of a 
signed distance function would even enforce |V@| = 1). 

The Dirac delta could as well be replaced by a positive constant, say 1, which 
yields another choice for a descent direction 


F(x) = — Re(be—b;) grad, ZY forall xe Q. (152) 
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This new direction F,,,(x) has the property that it applies updates driven by data 
sensitivities on the entire domain and thereby enables the creation of objects far 
away from the actual zero level set by lowering a positive level set function until its 
values arrive at zero. Certainly, at this moment when the level set function changes 
at some points far away from the zero level set from positive to negative values, 
a new object is created, and the descent property with respect to the cost needs to 
be evaluated by introducing some concept evaluating the effect of object creation on 
the data misfit. A formal way of doing so is briefly discussed in section “Topological 
Derivatives” further below. The opposite effect that inside a given shape, with some 
distance from the zero level set, the values of the level set function switch from 
negative to positive values can also occur, in which case a hole is created inside 
the shape. Also here, justification of this hole creation with respect to its effect on 
the data misfit cost is needed and can be treated as well by the tools discussed in 
section “Topological Derivatives.” 

Notice that, in the more classical level set framework, these replacements of 
the Dirac delta by functions with extended support can be interpreted as different 
ways of defining extension velocities for the numerical level set evolution scheme. 
Refer to [7, 19, 45, 47, 84] and the further references given there for numerical 
approaches which are focusing on incorporating topology changes during the shape 
reconstruction. 

Once optimization schemes are considered for level set-based shape reconstruc- 
tion, a rich set of classical optimization schemes can be adapted and applied to this 
novel application. For example, Newton-type optimization techniques and second- 
order shape derivatives can be defined and calculated. Strategies for doing so are 
available in the literature; see, for example, [30]. Also quasi-Newton-, Gauss— 
Newton-, or Levenberg—Marquardt-type schemes look promising in this framework. 
Some related approaches can be found, for example, in [18, 50, 82, 90, 93]. There 
exists a large amount of literature concerned with shape optimization problems 
in various applications. One important application is, for example, the structural 
optimal shape design problem, where the shape of a given object (a tool, bridge, tele- 
graph pole, airplane wing, etc.) needs to be optimized subject to certain application- 
specific constraints [3, 87,96]. Another example is the optimization of a bandgap 
structure or of maximal eigenvalues [46, 56,75]. Some techniques from nonlinear 
optimization which have been successful in those applications consequently have 
also found their way into the treatment of shape inverse problems. For brevity, we 
simply refer here to the discussions presented in [2, 18, 20, 43, 50, 82, 90, 93] and 
the many further references therein. Alternative nonlinear algebraic reconstruction 
techniques are employed in [34] and fixed point techniques in [22, 23]. 


Some Remarks on Numerical Shape Evolution with Level Sets 
Not much is said here regarding numerical schemes for solving Hamilton—Jacobi 


equations numerically or for solving the related optimality systems for shape inverse 
problems numerically. In the framework of imaging science, various schemes have 
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been developed and discussed extensively in the vast literature on numerical level 
set evolution; see, for example, the books and reviews [74, 85,91], to mention just a 
few examples. These schemes include CFL conditions, re-initialization of level set 
functions, signed distance functions, the fast marching method, higher-order upwind 
schemes like ENO (essentially non-oscillating) and WENO (weighted essentially 
non-oscillating), artificial viscosity solutions, numerical discretizations of mean 
curvature terms in the level set framework, etc. All these techniques can be applied 
when working on the treatment of inverse problems by a level set formulation. 

It is emphasized here, however, that the application of image reconstruction 
from indirect data comes with a number of additional problems and complications 
which are due to the ill-posedness of the inverse problem and to the often high 
complexity of the PDE (or IE) involved in the simulation of the data. Therefore, each 
particular image reconstruction problem from indirect data requires a careful study 
of numerical schemes which typically are tailor-made for the specific application. 
Overall, a careful choice of numerical discretization schemes and regularization 
parameters is indeed essential for a stable and efficient solution of the shape 
reconstruction problem. Moreover, also design parameters of the experimental setup 
(as, e.g., source and receiver locations) during the data collection have a significant 
impact on the shape evolution later on in the reconstruction process. Judicious 
choices here pay out in form of faster and more reliable reconstructions. 


Speed of Convergence and Local Minima 


Level set methods for shape reconstruction in inverse problems have initially 
been claimed to suffer from slow convergence due to inherent time-discretization 
constraints (the CFL condition) for the Hamilton—Jacobi equation and due to the (so 
far) exclusive use of first-order shape derivatives. Also, it had been observed that 
the shape evolution sometimes gets trapped in local minima, such that, for example, 
some topological components are missed by the shape evolution when starting with 
an inappropriate intitial guess. 

However, these initial problems seem to have been resolved by now, and it 
appears that level set methods have in fact become quite efficient and stable when 
following certain straightforward guidelines and often even clearly outperform 
many classical pixel-based reconstruction schemes when additional prior informa- 
tion is available. 

Firstly, the search for a good starting guess for the shape evolution can usually 
be done by either specific preprocessing steps (as, e.g., in [98]) or by employing 
more traditional search routines for only a few iteration steps. This helps avoiding 
“long-distance evolutions” during the succeeding shape reconstruction process. 

A similar effect is achieved by the incorporation of some form of “topological 
derivative” in the shape evolution algorithm; see the brief discussion of this topic 
in the following section “Topological Derivatives.” With this topological derivative 
technique, “seed” objects occur during the evolution just at the correct locations to 
be deformed in only few more iterations to their final shapes. 
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The topological derivative (or an appropriately designed extension velocity 
which has a similar effect) can also help in avoiding the shape evolution to become 
trapped in local minima due to barriers of low sensitivity where velocity fields 
become very small. Again by the effect of the creation of “seed” objects in areas 
of higher sensitivity, the shape evolution can jump over these barriers and quickly 
arrive at the final reconstruction. When an object is extended over an area of 
low sensitivity, then, certainly, any reconstruction scheme has difficulties with 
its reconstruction inside this zone, such that additional prior information might 
be needed for arriving at a satisfactory result inside this zone of low sensitivity 
(regardless which reconstruction technique is used). 

In addition, also higher-order shape derivatives have been developed in the 
literature (see, e.g., [30]) which can be used for deriving higher-order shape-based 
reconstruction schemes. So far, however, their usefulness as part of a level set-based 
shape inversion technique has been investigated only to a very limited extent. 

Finally, in an optimization framework, line-search techniques can replace the 
CFL condition for marching toward the sought minimum of a cost functional. This 
can speed up convergence significantly. 

Keeping these simple strategies in mind, level set-based reconstruction tech- 
niques can in fact be much faster than more traditional schemes, in particular when 
the contrast value of the parameters is assumed to be known and does not need to 
be recovered simultaneously with the shape. For very ill-posed inverse problems, 
traditional techniques need a large number of iterations to converge to the right 
balance between correct volume and contrast value of the sought objects. 


Topological Derivatives 


Even though the level set formulation allows for automatic topology changes during 
the shape evolution, the concepts on calculating descent directions derived so far do 
not really apply at the moment when a topological change occurs. This is typically 
no problem for the case of splitting and merging of shapes, since descent directions 
are only calculated for discrete time steps, such that practically always never the 
need arises to calculate a descent direction just when such a topological change 
occurs. Still, from a theoretical perspective, it would be interesting to calculate 
expressions also for topological derivatives which capture the splitting and merging 
of shapes. 

Another situation where topological changes occur in shape evolution is the 
creation and annihilation of shape components. These situations also occur auto- 
matically in the level set framework when a suitable extension velocity is chosen. 
However, for these two situations, explicit expressions have been derived in the 
literature which describe the impact of an infinitesimal topological change on the 
least squares data misfit cost. These are generally known as topological derivatives. 

The technique of topological derivatives has received much attention lately as 
a direct way of image reconstruction. The idea in these approaches is usually to 
calculate a value of the topological derivative (or topological sensitivity) at each 
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Q\D B, 


p+ dp 


Fig. 10 Creating a hole B, inside the shape D 


location of the imaging domain and then adding small geometric objects at those 
places where this topological sensitivity is the most negative. 

Certain issues arise here, as, for example, the question on how large these new 
objects should be, how close to each other or to the domain boundary they can 
be, and which contrast value should be applied for the so created small object. 
So far, in most cases, just one update in line with the above said is done, and 
thereafter the image reconstruction is either stopped or continued by a shape 
evolution of the so constructed set of objects. Nevertheless, the possibility of 
iterative topological reconstruction techniques remains an interesting challenge. 
Furthermore, the combination of simultaneous topological and shape evolution 
seems to be a very promising approach which combines the flexibility of level set 
evolution with the sensitivity driven creation and annihilation of shapes. This effect 
occurs in practice automatically if appropriate extension velocities are chosen in the 
regular level set shape evolution technique. 

In the following, a more formal approach to topological changes is presented 
which has the advantage of providing a stronger mathematical justification of 
topological changes in the goal of data misfit reduction. The discussion will be based 
on the general ideas described in the references [15,22,23, 39,40, 69,73, 83,86]. The 
topological derivative as described here aims at introducing either a small hole (let 
us call it B,) into an existing shape D or at adding a new object (let us call it D,) 
into the background material at some distance away from an already existing shape 
D (see Fig. 10). We will concentrate in the following on the first process, namely, 
adding a small hole into an existing shape. The complementary situation of creating 
a new shape component follows the same guidelines. 

Denote D, = D\B,, where the index p indicates the “size” of the hole B, and 
where it is assumed that the family of new holes defined by this index is “centered” 
at a given point x. (In other words, one has x € B, C B, for any0 < p < p! < 
1.) It is assumed that all boundaries are sufficiently smooth. Consider then a cost 
functional _Y (D) which depends on the shape D. The topological derivative Dr is 
defined as 
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9 = tim LD- £0) 
ee ee 


where f(p) is a function which approaches zero monotonically, i.e., f(o) — 0 for 
p — 0. With this definition, the asymptotic expansion follows 


J (Dp) = J(D) + f(p)Pr&) + o(f(p)). (154) 


Early applications of this technique (going back to [22, 23, 83]) were focusing on 
introducing ball-shaped holes into a given domain in connection to Dirichlet or 
Neumann problems for a Laplace equation. Here, the function f(e) is mainly deter- 
mined by geometrical factors of the created shape, and the topological derivative 
J (D,) can be determined by solving one forward and one adjoint problem for 
the underlying Laplace equation. In fact, for the Neumann problem for the Laplace 
equation using ball-shaped holes, the relationship (153) takes the original form 
introduced in [22,23,83] where f(p) is just the negative of the volume measure 
of the ball, ie., f() = —xp? in 2D and f(p) = —4:rp?/3 in 3D. For more details 
and examples, see [22]. In general, the details of the behavior of the limit in (153), 
as well as of the function f(p) if the limit exists, depend strongly on the shape of the 
hole, on the boundary condition at the hole interface, and on the underlying PDE. 

An attempt has been made recently to find alternative definitions for the 
topological derivative. One such approach has been presented in [39, 40,73]. Instead 
of taking the point of view that a hole is “created,” the topological derivative is 
modeled via a limiting process where an already existing hole gradually shrinks until 
it disappears. For example, perturb the parameter p of an existing hole by a small 
amount dp. Then, the cost Y (D,) is perturbed to Y (Dp+8p)s and the following 
limit appears: 


* _ : J (Dorin) = J (Dp) 
7) = Th anode + bp) — Fe) nen 


In [40, 73] the authors show a relationship between (153) and (155), which 
reads as 


Ir) = FEE) = lim 5; Oa —_—— 9x, (0), (156) 


where Wy, (p) is a specific form of a shape derivative related to a velocity flow V,, in 
the inward normal direction of the boundary 0B, with speed |V,,|. For more details, 
refer to [40,73]. A related link between shape derivative and topological derivative 
has been demonstrated also in [22]. Recently published-related work on this topic is 
briefly reviewed in [33]. 
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9 Case Studies 
Case Study: Microwave Breast Screening 


In section “Example |: Microwave Breast Screening,” a complex breast model is 
presented for tackling the problem of early breast cancer detection from microwave 
data. Due to the high complexity of the model, also the reconstruction algorithm 
is likely to show some complexity. In [52] a reconstruction technique is proposed 
which uses five consecutive stages for the reconstruction. In the first stage, a pixel- 
by-pixel reconstruction is performed for the interior fatty fibroglandular region, 
with the skin region being (at this stage of the algorithm typically still incorrectly) 
estimated and fixed. Once a pixel-based reconstruction has been achieved, an 
initial shape for the fibroglandular region (the background being then fatty tissue) 
is extracted from it, which then, in the succeeding stages, is evolved jointly 
with the interior profiles, the skin region, and a possible tumor region until the 
final reconstruction is achieved. An important feature of the algorithm is that in 
different stages of the algorithm, different combinations of the unknowns (level set 


Permittivity 


0 4 8 12 16 


Fig. 11 First breast model of Fig. 1 with a disk-shaped tumor of diameter 8 mm situated deeply 
inside the breast. Top left: reference permittivity profile (true tumor permittivity value ei" = 
53). Top center: the result at the end of stage I (pixel-by-pixel reconstruction). Top right: final 
reconstruction of level set-based structural inversion scheme (reconstructed permittivity value 
erconst’ = 50). Bottom: cross section through the correct tumor for constant y coordinate (the 
dashed line represents the true permittivity profile, the solid line the pixel-by-pixel result, and the 


dash-dotted line the structural inversion result). For more details, see [52] 


524 O. Dorn and D. Lesselier 


y [cm] 


60 T T T T T T T 


A 
oO 


Permittivity 


ye) 
fo) 


Fig. 12 Second breast model of Fig. 1 with a large fibroglandular tissue and a disk-shaped tumor 
of 6mm diameter. The images are arranged as in Fig. 11. The real static permittivity of the tumor 


. . reconst . . : . 
is ev™' = 52 and the reconstructed one ise, = 49. See also the animated movie provided in 


[52] which shows the shape evolution for this example 


functions and interior parameter profiles) are evolved. For more details regarding 
the reconstruction algorithm, refer to [52]. 

Here, the pixel-by-pixel reconstructions of stage I of the algorithm and the 
final reconstructions using the complex breast model and a level set evolution are 
presented for the three breast models introduced in Fig. 1 and compared with each 
other in the cases where a small tumor is present. See Figs. 11-13. The upper left 
image of each figure shows the real breast, the central upper image shows the pixel- 
by-pixel reconstruction with our basic reconstruction scheme, and the upper right 
image shows the level set-based reconstruction using the complex breast model 
explained in section “Example |: Microwave Breast Screening.” The bottom images 
show cross sections through a horizontal line indicated in the upper row images and 
passing through the tumor locations for the three images. 

The data are created on a different grid than the one used for the reconstruction. 
The corresponding signal-to-noise ratio is 26 dB. Forty antennas are used as sources 
and as receivers, which are situated equidistantly around the breast. Microwave 
frequencies of 1, 2,3, 4, and 5 GHz are used for the illumination of the breast. 

Even though the pixel-by-pixel reconstruction scheme is not optimized here, a 
general problem of pixel-based reconstruction can be identified immediately from 
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Fig. 13 Third breast model of Fig. | with a region of fibroglandular tissue intermixed with adipose 
tissue. The hidden tumor is an ellipsoid of 5 x 6mm (lengths of principle axes). The images are 
displayed as in Fig. 11. The real static permittivity value of the tumor is e{'"°" = 50 and the 


reconst 


reconstructed one is €, = 42. For more details, see [52] 


the presented examples. The reconstructions tend to be oversmoothed, and the small 
tumor can hardly be identified from the pixel-based reconstruction. By no means 
it is possible to give any reliable estimate from these pixel-based reconstructions 
for the contrast of the interior tumor values to the fibroglandular or fatty tissue 
values of static relative permittivity. True, the level set reconstruction scheme takes 
advantage of the fact that it closely follows the correct model for breast tissue. On 
the other hand, this information is typically available (at least approximately) in 
breast screening applications, such that better estimates of the tumor characteristics 
can be expected when using such a level set-based complex breast model. This is 
confirmed in the three reconstructions shown in the upper right images of Figs. 1 1— 
13. For more details on this reconstruction scheme in microwave breast screening, 
and for an animated movie showing the image evolution, see [52]. 


Case Study: History Matching in Petroleum Engineering 


Figure 14 shows the situation described in section “Example 2: History Matching 
in Petroleum Engineering” of history matching from production data. The image 
is composed of one zone of overall (approximately) bilinear behavior (a trend) 
and another zone where the permeability is smoothly varying without any clearly 
identifiable trend. The reconstruction follows this model and evolves simultaneously 
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Fig. 14 Case study: history matching in reservoir engineering from production data. Left column 
from top to bottom: reference model, final reconstruction, and evolution of parameter values 1, B2, 
Bs of the bilinear trend model; right column from top to bottom: initial guess, evolution of the least 
squares data misfit and the initial (red solid), final (black dashed), and reference (black solid) total 
water production rate in m3/s (i.e., the true and estimated measurements). The complete evolution 
as an animated file and more details on the reconstruction scheme can be found in [35] 


the region boundaries (i.e., the describing level set function), the three expansion 
parameters of the bilinear profile in the sandstone lithofacie, as well as the smoothly 
varying interior permeability profile in the shale region. The initial guess (upper 
right image of the figure) is obtained from well-log measurements. The true image 
is displayed in the upper left image of the figure and the final reconstruction in 
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Fig. 15 Case study: crack reconstruction by an evolution of a thin shape. Top row from left 
to right: initial guess, reconstruction after | and 2 iterations. Middle row: after 5, 10, and 20 
iterations. Bottom row: final reconstruction after 25 iterations, real crack distribution, and evolution 
of least squares data misfit with iteration index. The noise level is indicated in the bottom right 
image by a horizontal dotted line. One iteration amounts to successive application of the updates 
corresponding to the data of each source position in a single-step fashion 


the center left image. The center right image shows the evolution of the data misfit 
cost during the joint evolution of all model unknowns, the lower left image shows 
the evolution of the three model parameters for the bilinear model in one of the 
regions, and the lower right image shows the initial, true, and final production rate 
profile over production time averaged over all boreholes (the data). A classical 
pixel-based reconstruction scheme typically is not able to use different models 
for the parameter profiles in the different regions and simultaneously reconstruct 
the sharp region interfaces. For more details on this reconstruction scheme for 
history matching in reservoir engineering, including animated movies for additional 
numerical examples, see [35]. 


Case Study: Reconstruction of Thin Shapes (Cracks) 


Figure 15 shows a situation of shape evolution for the reconstruction of thin shapes 
(cracks) as described in section “Example 3: Crack Detection.’ The numerical 
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model presented in section “A Modification of the Classical Level Set Technique 
for Describing Cracks or Thin Shapes” is used here, where both level set functions 
describing the crack are evolved simultaneously driven by the least squares data 
misfit term. It is seen clearly that also in this specific model, topological changes 
occur automatically, when marching from a single initital (and somewhat arbitrary) 
crack candidate (upper left image of the figure) towards the final reconstruction 
showing three different crack components (bottom left image of the figure). The 
true situation is displayed in the bottom middle image of the figure, which shows as 
well three crack components which roughly are at the same location and of similar 
shape as the reconstructed ones. The evolution of the data misfit cost over artificial 
evolution time is displayed in the bottom right image of the figure. For more details 
on this reconstruction scheme and additional numerical experiments, see [4]. 
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Cross-References 


Readers interested in the material presented in this chapter will also find interesting 
and relevant additional material in many other chapters of this handbook. Some 
additional numerical results using level set techniques can be found, for example, in 
the chapter on EIT. Many concepts relevant to specific implementations of level set 
techniques can be found, amongst others, in the following chapters. 
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Abstract 


The aim of this chapter is to review recent developments in the mathematical 
and numerical modeling of anomaly detection and multi-physics biomedical 
imaging. Expansion methods are designed for anomaly detection. They provide 
robust and accurate reconstruction of the location and of some geometric features 
of the anomalies, even with moderately noisy data. Asymptotic analysis of the 
measured data in terms of the size of the unknown anomalies plays a key 
role in characterizing all the information about the anomaly that can be stably 
reconstructed from the measured data. In multi-physics imaging approaches, 
different physical types of waves are combined into one tomographic process 
to alleviate deficiencies of each separate type of waves while combining their 
strengths. Multi-physics systems are capable of high-resolution and high-contrast 
imaging. Asymptotic analysis plays a key role in multi-physics modalities 
as well. 


1 Introduction 


Inverse problems in medical imaging are in their most general form ill-posed. 
They literally have no solution [59, 86]. If, however, in advance one has additional 
structural information or can supply missing information, then one may be able 
to determine specific features about what one wishes to image with a satisfactory 
resolution and accuracy. One such type of information can be that the imaging 
problem is to find unknown small anomalies with significantly different parameters 
from those of the surrounding medium. These anomalies may represent potential 
tumors at an early stage. 

Over the last few years, an expansion technique has been developed for the 
imaging of such anomalies. It has proven useful in dealing with many medical 
imaging problems. The method relies on deriving asymptotics. Such asymptotics 
have been investigated in the case of the conductivity equation, the elasticity 
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equation, the Helmholtz equation, the Maxwell system, the wave equation, the 
heat equation, and the (modified) Stokes system. A remarkable feature of this 
method is that it allows a stable and accurate reconstruction of the location and 
of some geometric features of the anomalies, even with moderately noisy data. This 
is because the method reduces the set of admissible solutions and the number of 
unknowns. It can be seen as a kind of regularization in comparison with (nonlinear) 
iterative approaches. 

Another promising technique for efficient imaging is to combine into one 
tomographic process different physical types of waves. Doing so, one alleviates 
deficiencies of each separate type of waves while combining their strengths. Again, 
asymptotic analysis plays a key role in the design of robust and efficient imaging 
techniques based on this concept of multi-waves. In the last decade or so, work 
on multi-physics imaging in biomedical applications has come a long way. The 
motivation is to achieve high-resolution and high-contrast imaging. 

The objective of this chapter is threefold: (1) to provide asymptotic expansions 
for both internal and boundary perturbations that are due to the presence of small 
anomalies, (2) to apply those asymptotic formulas for the purpose of identifying 
the location and certain properties of the shape of the anomalies, and (3) to design 
efficient inversion algorithms in multi-physics modalities. 

Applications of the anomaly detection and multi-physics approaches in medical 
imaging are described in some detail. In particular, the use of asymptotic analysis to 
improve a multitude of emerging imaging techniques is highlighted. These imaging 
modalities include electrical impedance tomography, ultrasound imaging, infrared 
thermography, magnetic resonance elastography, impediography, magneto-acousto- 
electrical tomography, magneto-acoustic tomography with magnetic induction, and 
photo-acoustic imaging. They can be divided into three groups: (1) those using 
boundary or scattering measurements such as electrical impedance tomography, 
ultrasound, and infrared tomographies; (2) those using internal measurements such 
as magnetic resonance elastography; and (3) those using boundary measurements 
obtained from internal perturbations of the medium such as impediography and 
magneto-acoustic imaging. 

As it will be shown in this chapter, modalities from group (1) can only be used 
for anomaly detection, while those from groups (2) and (3) can provide a stable 
reconstruction of a distribution of physical parameters. 


2 Electrical Impedance Tomography for Anomaly Detection 
Physical Principles 


Electrical impedance tomography uses low-frequency electric current to probe a 
body; the method is sensitive to changes in electrical conductivity. By injecting 
known amounts of current and measuring the resulting electrical potential field 
at points on the boundary of the body, it is possible to “invert” such data to 
determine the conductivity or resistivity of the region of the body probed by the 
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currents. This method can also be used in principle to image changes in dielectric 
constant at higher frequencies, which is why the method is often called “impedance” 
tomography rather than “conductivity” or “resistivity” tomography. However, the 
aspect of the method that is most fully developed to date is the imaging of 
conductivity/resistivity. Potential applications of electrical impedance tomography 
include determination of cardiac output, monitoring for pulmonary edema, and in 
particular screening for breast cancer. 

Recently, a commercial system called TS2000 (Mirabel Medical Systems Inc., 
Austin, TX) has been released for adjunctive clinical uses with X-ray mammography 
in the diagnostic of breast cancer. The mathematical model of the TransScan can 
be viewed as a realistic or practical version of the general electrical impedance 
system. In the TransScan, a patient holds a metallic cylindrical reference electrode, 
through which a constant voltage of 1—2.5 V, with frequencies spanning 100 Hz— 
100 KHz, is applied. A scanning probe with a planar array of electrodes, kept at 
ground potential, is placed on the breast. The voltage difference between the hand 
and the probe induces a current flow through the breast, from which information 
about the impedance distribution in the breast can be extracted. 

The use of asymptotic analysis yields a rigorous mathematical framework for the 
TransScan. See [30,88] for a detailed study of this electrical impedance tomography 
system. 


Mathematical Model 


Let Q be a smooth bounded domain in R“, d = 2 or3 and let v, denote the outward 
normal to dQ at x. Suppose that the conductivity of Q is equal to 1. Let D denote 
a smooth anomaly inside Q with conductivity 0 < k # 1 < +00. The voltage 
potential in the presence of the set D of conductivity anomalies is denoted by u. It 
is the solution to the conductivity problem 


V-(x(Q\ D)+kx(D)) Vu=0 ing, 


a Be (« E 12009), f gdo = 0). (1) 
dQ aQ 


av 
udo =0, 
aQ 


where y(D) is the characteristic function of D. 
The background voltage potential U in the absence of any anomaly satisfies 


AU =0 in, 

dU 

Ole (2) 
Udo = 0. 


dQ 
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Let N(x, z) be the Neumann function for —A in Q corresponding to a Dirac mass 
at z. That is, NV is the solution to 


—A,N(x,z) = 6, inQ, 


oN 1 
eel | eee N = fe Q. 
ap bs a 73 (x,z)do(x) =0 forze 


(3) 


Note that the Neumann function N(x, z) is defined as a function of x € Q for each 
fixedz€ Q. 

For B a smooth bounded domain in R¢ and 0 < k # 1 < +00 a conductivity 
parameter, let ) = >(B, k) be the solution to 


Av=0 inR’\B, 


Av=0 inB, 
d-->|4 =O ondB, 

aol) oe (4) 
k—| —=—]| =0 onoB, 

av Ov fy 


v(€) —E +0 as |E| > +00. 


Here, one denotes 
vj4(&) = lim, v (§ + tye), & € OB, 
t—>0 


and 


“| @= 


Ove | 4 (Vv (é 7 tve) : ve), E € OB, 


lim 
toot 


if the limits exist, where vg is the outward unit normal to 0B at & and (, ) is the scalar 
product in R¢. For ease of notation, the dot will be sometimes used for the scalar 
product in R¢. 

Recall that > plays the role of the first-order corrector in the theory of homoge- 
nization [79]. 


Asymptotic Analysis of the Voltage Perturbations 


In this section, an asymptotic expansion of the voltage potentials in the presence 
of a diametrically small anomaly with conductivity different from the background 
conductivity is provided. 

The following theorem gives asymptotic formulas for both boundary and internal 
perturbations of the voltage potential that are due to the presence of a conductiv- 
ity anomaly. 
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Theorem 1 (Voltage perturbations). Suppose that D = 6B + z,6 being the 
characteristic size of D, and let u be the solution of (1), where0 <k #1 < +00. 
Denote by U the background solution, that is, the solution of (2). 


(i) The following asymptotic expansion of the voltage potential on 0Q holds for 
d = 2,3: 


u(x) & U(x) — 64 VU(z)M(k, B)V-N(x, z). (5) 


Here, N(x,z) is the Neumann function, that is, the solution to (3), and 
d 
M(k, B) = (mpq),,, 


, is the polarization tensor given by 


M(k, B) := k—1) [ Vo(E) dé, (6) 


where Y is the solution to (1). 
(ii) Let w be a smooth harmonic function in Q. The weighted boundary measure- 
ments I[U] satisfies 


1,[U] := / (u—-U) x) (x) do(x)  —64VU(z)-M(k, B)Vw(z). (7) 
aQ v 


(iii) The following inner asymptotic formula holds: 


xXx—Z 


) 


u(x) = U(2) + 89( )-vu@ for x near z. (8) 


The inner asymptotic expansion (8) uniquely characterizes the shape and the 
conductivity of the anomaly. In fact, suppose for two Lipschitz domains B and B’ 
and two conductivities k and k’ that 0(B,k) = v(B’,k’) in a domain englobing B 
and B’, then using the jump conditions satisfied by )(B,k) and »(B’,k’), one can 
easily prove that B = B’ andk = k’. 

The asymptotic expansion (5) expresses the fact that the conductivity anomaly 
can be modeled by a dipole far away from z. It does not hold uniformly in Q. It 
shows that, from an imaging point of view, the location z and the polarization tensor 
M of the anomaly are the only quantities that can be determined from boundary 
measurements of the voltage potential, assuming that the noise level is of order 
64+! Ttis then important to precisely characterize the polarization tensor and derive 
some of its properties, such as symmetry, positivity, and isoperimetric inequalities 
satisfied by its elements, in order to develop efficient algorithms for reconstructing 
conductivity anomalies of small volume. 

Some important properties of the polarization tensor are listed in the next 
theorem. 
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Theorem 2 (Properties of the polarization tensor). For0 < k #1 < +m, 
let M = M(k,B) = (mpq)4 


pq=l 
bounded domain B in R¢ and the conductivity k. Then 


be the polarization tensor associated with the 


(i) M is symmetric. 
(ii) [fk > 1, then M is positive definite, and it is negative definite if 0< k <1. 
(iii) The following isoperimetric inequalities for the polarization tensor 


1 
k-1 
(k — 1) trace(M™') < 


1 
trace(M) < («- 1+ :) |B, 
(9) 
d-1+k 
|B)” 


hold, where trace denotes the trace of a matrix and |B| is the volume of B. 


The polarization tensor M can be explicitly computed for disks and ellipses in the 
plane and balls and ellipsoids in three-dimensional space. See [25, pp. 81-89]. The 
formula of the polarization tensor for ellipses will be useful here. Let B be an ellipse 
whose semiaxes are on the x;- and x2-axes and of lengths a and ), respectively. 
Then, M(k, B) takes the form 


a+b 
M(k, B) = (k —1)|B| | 4+ *2 (10) 
0 


a+b 
b+ka 


Formula (5) shows that from boundary measurements, one can always represent 
and visualize an arbitrary-shaped anomaly by means of an equivalent ellipse of 
center z with the same polarization tensor. Further, it is impossible to extract 
the conductivity from the polarization tensor. The information contained in the 
polarization tensor is a mixture of the conductivity and the volume. A small anomaly 
with high conductivity and a larger anomaly with lower conductivity can have the 
same polarization tensor. 

The bounds (9) are known as the Hashin—Shtrikman bounds. By making use of 
these bounds, size and thickness estimations of B can be obtained. An inclusion 
whose trace of the associated polarization tensor is close to the upper bound must 
be infinitely thin [40]. 


Numerical Methods for Anomaly Detection 
In this section, one applies the asymptotic formula (5) for the purpose of identifying 


the location and certain properties of the shape of the conductivity anomalies. Two 
simple fundamental algorithms that take advantage of the smallness of the anoma- 


542 H. Ammari and H. Kang 


lies are singled out: projection-type algorithms and multiple signal classification 
(MUSIC)-type algorithms. These algorithms are fast, stable, and efficient. 


Detection of a Single Anomaly: A Projection-Type Algorithm 
One briefly discusses a simple algorithm for detecting a single anomaly. The reader 
can refer to [31,73] for further details. The projection-type location search algorithm 
makes use of constant current sources. One wants to apply a special type of current 
that makes VU constant in D. The injection current g = a- v fora fixed unit vector 
a € R¢ yields VU =a inQ. 

Assume for the sake of simplicity that d = 2 and D is a disk. Set 


w(y) = —(1/2m) log|x — y|  forx € R?\Q,y EQ. 
Since w is harmonic in Q, then from (7) to (10), it follows that 


. &-1)|D] @-D-4 


WIS e+) eae 


» SERN, (11) 


The first step for the reconstruction procedure is to locate the anomaly. The 
location search algorithm is as follows. Take two observation lines X; and YX 
contained in R?\Q given by 

x := aline parallel to a, 


Xi := aline normal toa. 
Find two points P; € &;,i = 1,2, so that 


Ty[U](Pi) = 9, Iw[U](P2) = max |f[U]9| 


From (11), one can see that the intersecting point P of the two lines 


T1(P1) = {x | a-(% — Pi) = 0}, (12) 
TIo(P2) := {x | (x — P2) is parallel to a} (13) 


is close to the center z of the anomaly D : |P — z| = O(8°). 

Once one locates the anomaly, the factor |D|(k — 1)/(k + 1) can be estimated. 
As it has been said before, this information is a mixture of the conductivity and the 
volume. A small anomaly with high conductivity and larger anomaly with lower 
conductivity can have the same polarization tensor. 

An arbitrary-shaped anomaly can be represented and visualized by means of an 
ellipse or an ellipsoid with the same polarization tensor. See Fig. 1. 
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Fig. 1 Detection of the location and the polarization tensor of a small arbitrary-shaped anomaly 
by a projection-type algorithm. The shape of the anomaly is approximated by an ellipse with the 
same polarization tensor 


We refer the reader to [57] for a discussion on the limits of the applicability of 
the projection-type location search algorithm and the derivation of a second efficient 
method, called the effective dipole method. 


Detection of Multiple Anomalies: A MUSIC-Type Algorithm 
Consider m well-separated anomalies D, = 5B, + z; (these are a fixed distance 


apart), with conductivities k;, s = 1,...,m. Suppose for the sake of simplicity that 
all the domains B, are disks. Let y; € R*\Q for] = 1,...,n denote the source 
points. Set 

Uy, = wy, := —(1/2)log|x—yi| forx€Q, J=1,...,n. 


The MUSIC-type location search algorithm for detecting multiple anomalies is as 
follows. For n € N sufficiently large, define the response matrix A = (A)’)/',_, by 


ft) 


Aig = Ty, (Cy = y F (u — Uy,,)(x) (x) da(x) . 


a 
Expansion (7) yields 
=“. 2(ks — 1)|Ds| 
Aly & — PS erage (zs) VUy, (25) . 
s=1 : 
Introduce 


gO) (U)acuUy()) 


where v* denotes the transpose of the vector v. 
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Lemma 1 (MUSIC characterization of the range of the response matrix). 
There exists ng > dm such that for any n > no, the following characterization 
of the location of the anomalies in terms of the range of the matrix A holds: 


g(x) € Range(A) if and only if x € {z,,...,Zn} . (14) 


The MUSIC-type algorithm to determine the location of the anomalies is as 
follows. Let Pnoise = 1 — P, where P is the orthogonal projection onto the range 
of A. Given any point x € , form the vector g(x). The point x coincides with the 
location of an anomaly if and only if Proiseg(x) = 0. Thus, one can form an image 
of the anomalies by plotting, at each point x, the cost function 


1 


Wy = ——., 
mel || PaoiseS (X)]|| 


The resulting plot will have large peaks at the locations of the anomalies. 
Once one locates the anomalies, the factors |D,|(k; —1)/(ks +1) can be 
estimated from the significant singular values of A. 


Bibliography and Open Questions 


Part (i) in Theorem | was proven in [21, 44,51]. The proof in [21] is based on a 
decomposition formula of the solution into a harmonic part and a refraction part 
first derived in [61]. Part (iii) is from [28]. The Hashin—Shtrikman bounds for the 
polarization tensor were proved in [43,77]. The projection algorithm was introduced 
in [31,73]. The MUSIC algorithm was originally developed for source separation 
in signal theory [94]. The MUSIC-type algorithm for locating small conductivity 
anomalies from the response matrix was first developed in [38]. The strong relation 
between MUSIC and linear sampling methods was clarified in [16]. The results of 
this section can be generalized to the detection of anisotropic anomalies [60]. 

As it has been said before, it is impossible to extract separately from the detected 
polarization tensor information about the material property and the size of the 
anomaly. However, if the measurement system is very sensitive, then making use 
of higher-order polarization tensors yields such information. See [25] for the notion 
of the higher-order polarization tensors. 

One of the most challenging problems in anomaly detection using electrical 
impedance tomography is that in practical measurements, one usually lacks exact 
knowledge of the boundary of the background domain. Because of this, the 
numerical reconstruction from the measured data is done using a model domain 
that represents the best guess for the true domain. However, it has been noticed that 
an inaccurate model of the boundary causes severe errors for the reconstructions. An 
elegant and original solution toward eliminating the error caused by an incorrectly 
modeled boundary has been proposed and implemented numerically in [69]. As 
nicely shown in [67], another promising approach is to use multifrequency data. 
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The anomaly can be detected from a weighted frequency difference of the measured 
boundary voltage perturbations. Moreover, this method eliminates the need for 
numerically simulated background measurements at the absence of the conductivity 
anomaly. See [58,67]. 


3 Ultrasound Imaging for Anomaly Detection 
Physical Principles 


Ultrasound imaging is a noninvasive, easily portable, and relatively inexpensive 
diagnostic modality which finds extensive clinical use. The major applications 
of ultrasound include many aspects of obstetrics and gynecology involving the 
assessment of fetal health, intra-abdominal imaging of the liver and kidney, and 
the detection of compromised blood flow in veins and arteries. 

Operating typically at frequencies between | and 10 MHz, ultrasound imaging 
produces images via the backscattering of mechanical energy from interfaces 
between tissues and small structures within tissue. It has high spatial resolution, 
particularly at high frequencies, and involves no ionizing radiation. The weaknesses 
of the technique include the relatively poor soft tissue contrast and the fact that 
gas and bone impede the passage of ultrasound waves, meaning that certain organs 
cannot easily be imaged. However, ultrasound imaging is a valuable technique for 
anomaly detection. It can be done in the time domain and the frequency domain. 

Mathematical models for acoustical soundings of biological media involve the 
Helmholtz equation in the frequency domain and the scalar wave equation in the 
time domain. 


Asymptotic Formulas in the Frequency Domain 


Let k and p be positive constants. With the notation of section “Asymptotic Analysis 
of the Voltage Perturbations,” p is the compressibility of the anomaly D and k is its 
volumetric mass density. The scalar acoustic pressure u generated by the Neumann 
data g in the presence of the anomaly D is the solution to the Helmholtz equation: 


V- (x (Q\D) + kx(D)) Vu + w(x (Q\D) + px(D)u =0 ind, 

du (15) 
—=g onoQ, 

ov 


while the background solution U satisfies 


AU +o0*U =0 inQ, 


a (16) 
= =g ondaQ. 
dv 
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Here, w is the operating frequency. A relevant boundary data g is the normal 
derivative of a plane wave e’®*"®, with the wavelength A := 277/a, traveling in the 
direction of the unit vector 6. 

Introduce the Neumann function for — (A + wo”) in Q corresponding to a Dirac 
mass at z. That is, N,, is the solution to 


— (A, + 0”) No(x,z) = 8, inQ, 


(17) 
aul lag =O ondQ. 
Ovx 


Assuming that w? is not an eigenvalue for the operator —A in L?(Q) with 
homogeneous Neumann boundary conditions, one can prove, using the theory of 
relatively compact operators, existence and uniqueness of a solution to (15) at least 
for 6 small enough [95]. Moreover, the following asymptotic formula holds. 


Theorem 3 (Pressure perturbations). Let u be the solution of (15) and let U be 
the background solution. Suppose that D = 5B + z, withO < (k,p) # (1,1) < 
+oo. Suppose that wb < 1. 

(i) For any x € 0Q, 


u(x) & U(x) -— a (VU(z)-M(k, B)V.Nw(Xx, z) 
+0*(p— 1)|B|U@)No(x.2)). (18) 
where M(k, B) is the polarization tensor associated with B and k. 


(11) Let w be a smooth function such that (A + wo”) w = 0 in Q. The weighted 
boundary measurements I,,[U, w] satisfy 


0 
1,[U,0] = / {4 UN) 5") a(x) 


~ —54 (VU(z)- M(k, B)Vw(z) + @7(p — 1) B|U(z)w(2)) . 
(19) 


(iii) The following inner asymptotic formula holds: 


Ko % 


u(x) = U(z) + 5% ( ) -VU(z)_ for x near z, (20) 


where ¥ is the solution to (1). 


Compared to the conductivity equation, the only extra difficulty in establishing 
asymptotic formulas for the Helmholtz equation (15) as the size of the acoustic 
anomaly goes to zero is that the equations inside and outside the anomaly are not 
the same. 
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Asymptotic Formulas in the Time Domain 


Suppose that o = 1. Consider the initial boundary value problem for the (scalar) 
wave equation 


dru—V-(x(Q\D) +kx(D)) Vu=0 in Qr, 
u(x,0) = uo(x), d;,u(x,0) = u(x) forx €Q, 


(21) 
ou 
—=g on 0Qr7, 
dv 
where T < -+o0 is a final observation time, Qr = Qx]0,7T[,and 0DQr = 


dQx]0, T[. The initial data ug,u) € C% (Q) and the Neumann boundary data 
g €C~(0, T;C~(dQ)) are subject to compatibility conditions. 
Define the background solution U to be the solution of the wave equation in the 
absence of any anomalies. Thus, U satisfies 
dU — AU =0 in Qr, 
U(x,0) = uo(x), 0,U(x,0) = u(x) forx €Q, 


0 
a =g ondQr. 
ov 


For p > 0, define the operator P, on tempered distributions by 


PpLW] (x1) = / oe V=Tet h(x, w) de, (22) 


lol<p 
where w(x, @) denotes the Fourier transform of w(x, ft) in the t-variable. Clearly, 
the operator P, truncates the high-frequency component of y. 
The following asymptotic expansion holds as 6 — 0. 
Theorem 4 (Perturbations of weighted boundary measurements). Let w € 


Ce (Qr) satisfy (0? - A) w(x,t) = 0 in Qr with 0,w(x,T) = w(x,T) = 0 
for x € Q. Suppose that p « 1/6. Define the weighted boundary measurements 


1,[U, T] := I Plu — U}o.0 0.) do(x) dat. 


Then, for any fixed T > diam(Q), the following asymptotic expansion for I,,[U, T] 
holds as 5 — 0: 


T 
LAT ~ tf VP[U](z, t)M(k, B)Vw(z, t) dt, (23) 


where M(k, B) is defined by (6). 
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Expansion (23) is a weighted expansion. Pointwise expansions similar to those 
in Theorem | which is for the steady-state model can also be obtained. 
Let y € R® be such that | y — z| >> 5. Choose 


br=|x—y| 


U(x,t) := U,(x,t) = forx # y. (24) 


4n|x — y| 
It is easy to check that U,, is the outgoing Green function to the wave equation: 
(a7 — A) Uy (x,t) = 5:=y8;=0 in R’x]0, +00. 


Moreover, Uy, satisfies the initial conditions: U,(x,0) = 0,U,(x,0) = 0 forx # 
y. Consider now for the sake of simplicity the wave equation in the whole three- 
dimensional space with appropriate initial conditions: 


dru—V- (x (R°\D) + kx(D)) Vu = b=y8;=0 in R*x]0, +00, 


(25) 
u(x,0)=0, d,u(x,0)=0 forxe R?,x xy. 


The following theorem holds. 


Theorem 5 (Pointwise perturbations). Let u be the solution to (25). Set Uy to be 
the background solution. Suppose that p « 1/6. 


(i) The following outer expansion holds 


P,[u— Uy] (x,t) © -8 | VP [UJ](x,t — t)-M(k, B)VP,[U,](z, t) dt , 
R 


(26) 
for x away from z, where M(k, B) is defined by (6) and Uy and U, by (24). 
(ii) The following inner approximation holds: 


X—Z 


Plu — Uy|(x, t) © 50 ( ) -VP,[Uy|(x.t) for x near z, (27) 


where ¥ is given by (4) and U, by (24). 


Formula (26) shows that the perturbation due to the anomaly is in the time 
domain a wave front emitted by a dipolar source located at point z. 

Taking the Fourier transform of (26) in the time variable yields the expansions 
given in Theorem 3 for the perturbations resulting from the presence of a small 
anomaly for solutions to the Helmholtz equation at low frequencies (at large 
wavelengths compared to the size of the anomaly). 
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Numerical Methods 


MUSIC-Type Imaging at a Single Frequency 

Consider m well-separated anomalies D, = 6B; + z5,s = 1,...,m. The 
compressibility and volumetric mass density of D, are denoted by p, and k,, 
respectively. Suppose as before that all the domains B, are disks. Let (@,..., 6,) be 
n unit vectors in R¢. For arbitrary 6 € {6,...,9,}, one assumes that one is in the 
possession of the boundary data u when the object Q is illuminated with the plane 
wave U(x) = e!9°*. Therefore, taking w(x) = e7!@"* for 6’ € {01,...,On} 
shows that one is in possession of 


Yip AQ Se 


for 0,0’ € {0,,...,0,}. Define the response matrix A = (Ay )} pay € C™ by 


6. P+ (p= i))e es 


. (ks — 1) iw(0)—0)7) *z 
Aw =) D,| (2 6, + Or al pce cae I A eh 
Ul 2 mn Rer + (ps —1)) Je n 
Introduce 


g(x) — (C1, A)* eho = a aa On) ein) 


Analogously to Lemma |, one has the following characterization of the location 
of the anomalies in terms of the range of the matrix A. 


Lemma 2 (MUSIC characterization of the range of the response matrix). 


There exists no € N,no > (d + 1)m, such that for any n > no, the following 
statement holds: 


g/(x) € Range(A) ifand only if x €{a,....2n} forj =1,..., d+, 
where g/ (x) is the jth column of g(x). 
The MUSIC algorithm can now be used as before to determine the location of 


the anomalies. Let Phoise = J — P, where P is the orthogonal projection onto the 
range of A. The imaging functional 


1 
Wau (x) = 7d ; 
a || PaoiseS/ (x)|| 


has large peaks only at the locations of the anomalies. See Fig. 2. 
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Fig. 2 MUSIC-type reconstruction from the singular value decomposition of A represented in 


Fig. 3 
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The significant singular vectors of A can be computed by the singular value 
decomposition. The number of significant singular values determines the number 
of anomalies. If, for example, k; # 1 and p, ~ 1 forall s = 1,...,m, then there 
are exactly (d + 1)m significant singular values of A and the rest are zero or close 
to zero. See Fig.3. The significant singular values of A can be used to estimate 


ks 
eo |Ds| and (ps ~ 1)|Ds]. 


Backpropagation-Type Imaging at a Single Frequency 
A backpropagation imaging functional at a single frequency w is given by 


n 


1 —2iw0) +x 
Wep(x) = — De Do Ui, 
/=1 
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where U(x) = wi(x) = e!@%"* for 6; € {6),...,4,} . Suppose that (0;,..., 4) 
are equidistant points on the unit sphere S“~!. For sufficiently large n, since 


1 J ela jo(2o|x|) ford = 3, 
n Jo(2@|x|) ford = 2, 


where jo is the spherical Bessel function of order zero and Jp is the Bessel function 
of the first kind and of order zero, it follows that 


= =) Jo(2@|x —zs|) ford = 3 
W, ~ Ds s—l : ; 
r(x) © 2! | Ge k, +1 + fe ) Jo(2@|x —z5|) ford = 2. 
An analogy between the backpropagation and MUSIC-type imaging can be 
established. Suppose that k, = 1 for s = 1,...,m. One can see that 
1 
Wau(x) o 


|Ds|(es — 1) — Wep(x) 


for x near z,; [9]. 


Kirchhoff-Type Imaging Using a Broad Range of Frequencies 
Let y, € R?\Q for/ = 1,...,n denote an array of source points. Set 


i 
Wy, (x) = Ho (lx —yil) and Uy,(x) = SHO (olx — yr), 
where Hi ") is the Hankel function of first kind and order zero. Using the asymptotic 


form of the Hankel function, one finds that for w|x — y| > 1, 


i ei t/4 
~H (w|x — yl) © eens 
a 2V2n Jolx — yl 


and 


VAL (|x —y|)x 


a iw@(x _ y) e ‘i eielx—yl 
q : 


1 
2/20 ( |x — y| Vo|x — yl 


Assume a high-frequency regime with wL > 1 for L the distance from the array 
center point to the locations z,,s = 1,...,m. It follows that 


Ty, (Ui, @] « 


m 


3 ID, (-2; ~1G@ = y) Gy) Be ») eiallzs—yil+lzs—yir) 
ks +1 [zs — yilles — yr 


552 H. Ammari and H. Kang 


Introduce the response matrix A(w) = (A; (@)) by 
Ajy/(@) = Ly, [Ur a] 


and the illumination vector 


g(x,0) = ((1. | eile, (1 a2)" golem) 
Ix yi] |x — yal 


In the case of measurements at multiple frequencies (w; ), we construct the weighted 
Kirchhoff imaging functional as 


1 
Mae =F DY) di (se, 0;),u@;)) (g@.0;).71(@;)). 
Jol 


where (a,b) = a@-b, J is the number of frequencies, and u; and v; are, respectively, 
the left and right singular vectors of A. As for Wyu, Wx is written in terms of the 
singular-value decompositions of the response matrices A(w;). 


Time-Reversal Imaging 
Unlike the three previous imaging methods, the one in this section is in time domain. 
It is based on time reversal. 

The main idea of time reversal is to take advantage of the reversibility of the 
wave equation in a non-dissipative unknown medium in order to back propagate 
signals to the sources that emitted them. In the context of anomaly detection, one 
measures the perturbation of the wave on a closed surface surrounding the anomaly 
and retransmits it through the background medium in a time-reversed chronology. 
Then the perturbation will travel back to the location of the anomaly. One can show 
that the time-reversal perturbation focuses on the location z of the anomaly with a 
focal spot size limited to one-half the wavelength which is in agreement with the 
Rayleigh resolution limit. 

In mathematical terms, suppose that one is able to measure the perturbation u — 
U,, and its normal derivative at any point x on a sphere S englobing the anomaly 
D and for a large time f9. The time-reversal operation is described by the transform 
t +» fo — t. Both the perturbation and its normal derivative on S are time reversed 
and emitted from S. Then a time-reversed perturbation propagates inside the volume 
surrounded by S. 

To detect the anomaly from measurements of the wavefield u—U,, away from the 
anomaly, one can use a time-reversal technique. Taking into account the definition of 
the outgoing fundamental solution (24) to the wave equation, spatial reciprocity, and 
time-reversal invariance of the wave equation, one defines the time-reversal imaging 
functional Wrr by 
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OP, |u— Uy 
Wrr(x, t) = [I [e. (x’,t —s) a (x’,t9 — 8) 


- (x’.t —s) Py [u = Uy (x’, to — »)| do (x’) ds, 


(28) 


where 


The imaging functional Wrpg corresponds to propagating inside the volume sur- 
rounded by S,, the time-reversed perturbation P,[u — U,], and its normal derivative 
on S. Theorem 5 shows that 


P,[u—U,] (x,t) » —8 i: V P,[U](x,t — t)-m(z,t) dt, 
where 
m(z,t) = M(k, B)VP,[U,](z, t) . (29) 
Therefore, since 


hhc oe le OF NG i 9) 


ero — s)Pp[Uz|(x', to — 5 — 0] da(x') ds 
dv 


= P,[U-](x, to —t — t) — P,[U\(x,t —t +17), 


(30) 


one obtains the approximation 
Wrr(x,t) » —8 / m(z, Tt): Vz | Po[U-](x, to —t—t)—P,[U|(x,t —to + t)|dt 
R 


which can be interpreted as the superposition of incoming and outgoing waves, 
centered on the location z of the anomaly. Since 


sin p(t — |x — y|) 
2n(t — |x — y|)|x — yl’ 


P[Uy](, 7) = 


m(z, T) is concentrated at the travel time t = T = |z— y]. It then follows that 


Wrr(x,t) © —83m(z, T)- Vz [ Po[U-l(x, to — T —t) — Pp[U-|(x,t — to + T)]. 
(31) 
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The imaging functional Wye is clearly the sum of incoming and outgoing polarized 
spherical waves. 

Approximation (31) has an important physical interpretation. By changing the 
origin of time, T can be set to 0 without loss of generality. Then by taking a Fourier 
transform of (31) over the time variable f, one obtains that 


Wrr(x, @) « 83m(z,T)+ V jo(w|x — zl), 


where w is the wave number. This shows that the time-reversal perturbation Wrp 
focuses on the location z of the anomaly with a focal spot size limited to one-half 
the wavelength. 

An identity parallel to (30) can be derived in the frequency domain. In fact, one 
has 


2 ’ av. ’ dU NT (¥! ’ en ad 
[ a, (x’) ho (x’) -— iy (x’) U, (x | do (x') = 2i3m U(x) (32) 
& jo(w|x — zl), 


which shows that in the frequency domain, Wrpr coincides with Wpp. 


Bibliography and Open Questions 


The initial boundary-value problems for the wave equation in the presence of 
anomalies of small volume have been considered in [5, 24]. Theorem 5 is from 
[12]. In [12], a time-reversal approach was also designed for locating the anomaly 
from the outer expansion (26). The physics literature on time reversal is quite rich. 
One refers, for instance, to [48] and the references therein. See [93] for clinical 
applications of time reversal. Many interesting mathematical works have dealt 
with different aspects of time-reversal phenomena: see, for instance, [33] for time 
reversal in the time domain, [45—47, 82] for time reversal in the frequency domain, 
and [37,50] for time reversal in random media. 

The MUSIC-type algorithm for locating small acoustic or electromagnetic 
anomalies from the multi-static response matrix at a fixed frequency was developed 
in [18]. See also [18-20], where a variety of numerical results was presented to 
highlight its potential and its limitation. It is worth mentioning that the MUSIC- 
type algorithm is related to time reversal [82, 87]. 

MUSIC and Kirchhoff imaging functionals can be extended to the time domain 
in order to detect the anomaly and its polarization tensor from (dynamical) boundary 
measurements [24]. 

The inner expansion in Theorem 5 can be used to design an efficient optimization 
algorithm for reconstructing the shape and the physical parameter of an anomaly 
from the near-field perturbations of the wavefield, which can be used in radiation 
force imaging. 
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In radiation force imaging, one uses the acoustic radiation force of an ultrasonic 
focused beam to remotely generate mechanical vibrations in organs. A spatiotem- 
poral sequence of the propagation of the induced transient wave can be acquired, 
leading to a quantitative estimation of the physical parameters of the anomaly. See, 
for instance, [35,36]. 

The proposed location search algorithms using transient wave or broad-range 
multifrequency boundary measurements can be extended to the case with limited- 
view measurements. Using the geometrical control method [34], one can still exploit 
those algorithms and perform imaging with essentially the same resolution using 
partial data as using complete data, provided that the geometric optics condition 
holds. 

An identity similar to (32) can be derived in an inhomogeneous medium, which 
shows that the sharper the behavior of the imaginary part of the Green function 
around the location of the anomaly is, the higher is the resolution. It would be quite 
challenging to explicitly see how this behavior depends on the heterogeneity of the 
surrounding medium. This would yield super-resolved ultrasound imaging systems. 


4 Infrared Thermal Imaging 
Physical Principles 


Infrared thermal imaging is becoming a common screening modality in the area of 
breast cancer. By carefully examining the aspects of temperature and blood vessels 
of the breasts in thermal images, signs of possible cancer or precancerous cell 
growth may be detected up to 10 years prior to being discovered using any other 
procedure. This provides the earliest detection of cancer possible. 

Because of thermal imaging’s extreme sensitivity, these temperature variations 
and vascular changes may be among the earliest signs of breast cancer and/or a 
precancerous state of the breast. An abnormal infrared image of the breast is an 
important marker of high risk for developing breast cancer. See [3, 83]. 


Asymptotic Analysis of Temperature Perturbations 


Suppose that the background Q is homogeneous with thermal conductivity 1 and 
that the anomaly D = 6B + zhas thermal conductivity 0 < k 4 1 < +oo. In this 
section, one considers the following transmission problem for the heat equation: 


d,u—V- (xy(Q\D) + ky(D)) Vu=0 in Qr, 
u(x,0) = uo(x) forx € Q, 


me =g ondQr, 
ov 


(33) 
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where the Neumann boundary data g and the initial data up are subject to a 
compatibility condition. Let U be the background solution defined as the solution 
of 


0,U —AU =0 inQr, 


U(x,0) = uo(x) forx € Q, 


ue =g ondQr. 
dv 


The following asymptotic expansion holds as 6 — 0. 


Theorem 6 (Perturbations of weighted boundary measurements). Let w ¢€ 
Ce (Qr) be a solution to the adjoint problem, namely, satisfy (0;-+ A)w(x,t) = 0 
in Qr with w(x, T) = 0 for x € Q. Define the weighted boundary measurements 


1,[U, T] := i. (ue, (x, dow) dt. 


Then, for any fixed T > 0, the following asymptotic expansion for I,,[U, T| holds 
as5 > 0: 


T 
Git. Tle =" i; VU(z,t)- M(k, B)Vw(z, t) dt, (34) 
0 


where M(k, B) is defined by (6). 


Note that (34) holds for any fixed positive final time 7, while (23) holds only for 
T > diam(Q). This difference comes from the finite speed propagation property for 
the wave equation compared to the infinite one for the heat equation. 

Consider now the background solution to be the Green function of the heat 
equation at y: 


_ |x=yl? 


e Tt 
U(x,t) = Uy(x.1) = 4 Gane? fone en) (35) 
O fort <0. 


Let u be the solution to the following heat equation with an appropriate initial 
condition: 


du—V- (x (R4\D) +kx(D)) Vu =0 in R“x]0, +00, 
(36) 
u(x,0) = U,(x,0) forx é€ R¢. 
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Proceeding as in the derivation of (26), one can prove that éu(x,t) := u—U is 
approximated by 


1 _ lx’? OD x’ —z 
k=] a ee ae (7 - VU, (x’,t) do (x’) dt, 
( | acy ||, 4 =| ( 5 ) U, (x’, t) da (x’) dr 
(37) 


for x near z. Therefore, analogously to Theorem 5, the following pointwise 
expansion follows from the approximation (37). 


Theorem 7 (Pointwise perturbations). Let y € R¢ be such that |y — z| >> 8. Let 
u be the solution to (36). The following expansion holds 


(u—U)(x,t) ~ —84 i VU_(x,t—1t)M(k, B)VU,(z,t) dt for |x—z| > O(6), 
0 


(38) 
where M(k, B) is defined by (6) and Uy and U, by (35). 


When comparing (38) and (26), one should point out that for the heat equation 
the perturbation due to the anomaly is accumulated over time. 

An asymptotic formalism for the realistic half-space model for thermal imaging, 
well suited for the design of anomaly reconstruction algorithms, has been developed 
in [29]. 


Numerical Methods 


In this section, the formula (34) is applied (with an appropriate choice of test 
functions w and background solutions U) for the purpose of identifying the location 
of the anomaly D. The first algorithm makes use of constant heat flux, and, not 
surprisingly, it is limited in its ability to effectively locate multiple anomalies. 

Using many heat sources, one then describes an efficient method to locate 
multiple anomalies and illustrate its feasibility. For the sake of simplicity, only the 
two-dimensional case will be considered. 


Detection of a Single Anomaly 
For y € R*\Q, let 


1 lx—yl? 
w(x, t) = wy(x, ft) = rer ers laa (39) 


The function w satisfies (0; + A)w = 0 in Q, and the final condition w|;,z7 = 0 
in Q. 

Suppose that there is only one anomaly D = z+ 6B with thermal conductivity 
k. For simplicity, assume that B is a disk. Choose the background solution U(x, t) 
to be a harmonic (time-independent) function in Q7. One computes 
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y-z — ley 
Vwy(z,t) = BaF —12° aT) | 


(K-IB] y-z koe 


M(k, B)Vwy(z,t) = KEL 4a(P 5? e4T=9 , 


and 
: (k= 1IBly—z p? etn 
/ M(k, B)Vwy(z,t) dt = i dt. 
0 . k + 1 4n 0 (T _ t)? 
But 
k=yP ss —|z— y |? oars 
—_@ 47-5 — y 
dt 4 (T —t)? 


and therefore 


(kK-1)|B] yr-—z _lkP 
e 45, 
kK+1  x\z—yl? 


T- 
/ M(k, B)Vwy(z,t) dt = 
0 


Then the asymptotic expansion (34) yields 


1 VU(z)-(y = 2) =z? 
\B| 2) : 2) bee 
my —z| 


k- 
L,[U,T ~ & 
[U, T](y) aa 


(40) 


Now, one is in a position to present the projection-type location search algorithm 
for detecting a single anomaly. Prescribe the initial condition ug(x) = a- x for some 
fixed unit constant vector a and choose g = a-v as an applied time-independent 
heat flux on 0Q7, where a is taken to be a coordinate unit vector. Take two 


observation lines ©; and D> contained in R? \Q such that 
1 := a line parallel to a, Xo := aline normaltoa. 
Next, find two points P; € 5;(i = 1,2) so that J,,(T)(P,) = 0 and 
_ L(T)(x) ifk—-—1 <0, 


1.(T)(P2) = . 
max L(T)(x) ifk—-—1>0. 
x€h2 


Finally, draw the corresponding lines I1;(P;) and II2(P2) given by (12). Then the 
intersecting point Pof IT,(P,) MN T12(P2) is close to the anomaly D : |P —2z| = 


O(6 |log 6|) for 6 small enough. 
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Detection of Multiple Anomalies: A MUSIC-Type Algorithm 
Consider m well-separated anomalies D, = 6B, + z5,5 = 1,...,m, whose heat 
conductivity is k;. Choose 


Ixy! 7 


1 
U(x,t) = Uy (x,t) = rere a 


for y’ € R?\Q 


or, equivalently, g to be the heat flux corresponding to a heat source placed at the 
point source y’ and the initial condition uo(x) = 0 in Q, to obtain that 


(1 —ks) 
6472 


T 1 —z,| 12/2 
x | seh paar ly =a a 
9 ¢t2(T —t) 4(T —t) 4t 


where w is given by (39) and M“) is the polarization tensor of D,. 

Suppose for the sake of simplicity that all the domains B, are disks. Then it 
follows from (10) that M“) = m“) Jy, where m“) = 2(k, — 1)|Bs|/(ks + 1) and Ih 
is the 2 x 2 identity matrix. Let yy € R?\Q for/ € N be the source points. One 
assumes that the countable set {;};en has the property that any analytic function 
which vanishes in {y;};en vanishes identically. 

The MUSIC-type location search algorithm for detecting multiple anomalies is 
as follows. For n € N sufficiently large, define the matrix A = [A),']/,_, by 


1,[U,T] ~ —8 > (y' =z) M(y = 2) 
s=1 


m a = ks) . 
Ay 2= —8 -. En — 2s)* (Yi — 2s) 


s=1 
is 1 Iyi— zs? ly — 2s? 
x > exp | - - dt. 
9 ¢2(T —t)? 4(T —t) 4t 


For z € &2, one decomposes the symmetric real matrix C defined by 


T 1 a 3)2 ,—>l2 
om / dus _lyaae buat) a, 
9 02(T —t) 4(T —t) At LI/'=1..n 


as follows: 
Pp 
— Yo viz)" (41) 
1=1 


for some p <n, where v; € R" and v; denotes the transpose of v;. Define the 
vector g() € R"*? for z € Q by 
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8 = (1-21 @,---s On —DVin(D)*, P= 1... P- (42) 
Here, vj1,..., Vin are the components of the vector v;,/ = 1,..., p. Let yw) = 
(Yix, Vly) for? = 1,...,0,2 = (2, Zy), and z; = (Zsx, Zsy). One also introduces 


g) = (yx — zx) 2), --+ 5 nx — Zx)Vin(@))* 


and 


1 
2 = = 


(Qnty — 4), ---s Ony — zyVin@)” 

Lemma 3 (MUSIC characterization of the range of the response matrix). The 
following characterization of the location of the anomalies in terms of the range of 
the matrix A holds: 


( and g\) € Range(A) Vile {l,...,p} ifandonlyif z€ {z,...,Zn}. 
(43) 


Note that the smallest number 1 which is sufficient to efficiently recover the 
anomalies depends on the (unknown) number m. This is the main reason for taking 
n sufficiently large. As for the electrical impedance imaging, the MUSIC-type 
algorithm for the thermal imaging is as follows. Compute Proise, the projection onto 
the noise space, by the singular value decomposition of the matrix A. Compute the 
vectors vy by oe aie a image of the locations, z),...,Zm, by plotting, at each 
a / | Proise ( gl. a)| for] = 1,..., p, where g? is 
given by (42) and a is a unit constant Ce The resulting isi will have large 
peaks at the locations of z;,s5 = 1,...,m. 

The next two figures (Figs. 4 and 5) show MUSIC-type reconstructions of two 
anomalies without and with noise. 

In Fig. 4, one sees clearly the presence of two anomalies. However, the one on 
the right, which is also deeper, is not as well rendered as the one on the left. 


point z, 


Bibliography and Open Questions 


Thermal imaging of small anomalies has been considered in [17]. See also [29], 
where a realistic half-space model for thermal imaging was considered and accurate 
and robust reconstruction algorithms are designed. 

It is worth mentioning that the inner expansions derived for the heat equation 
can be used to improve reconstruction in ultrasonic temperature imaging. The idea 
behind ultrasonic temperature imaging hinges on measuring local temperature near 
anomalies. The aim is to reconstruct anomalies with higher spatial and contrast 
resolution as compared to those obtained from boundary measurements alone. 
Further numerical investigations on this emerging topic are required. 
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Fig. 4 Detection of anomalies using n = 10 heat sources equi-placed on the top 
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Fig. 5 Detection in the presence of 1 % (on the left) and 5 % (on the right) of measurement noise 


5 Impediography 
Physical Principles 


Since all the present electrical impedance tomography technologies are only practi- 
cally applicable in feature extraction of anomalies, improving electrical impedance 
tomography calls for innovative measurement techniques that incorporate struc- 
tural information. A very promising direction of research is the recent magnetic 
resonance imaging technique, called current density imaging, which measures the 
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internal current density distribution. See the breakthrough work by Seo and his 
group described, for instance, in [65, 66, 89]. However, this technique has a number 
of disadvantages, among which the lack of portability and a potentially long imaging 
time. Moreover, it uses an expensive magnetic resonance imaging scanner. 

Impediography is another mathematical direction for future electrical impedance 
tomography research in view of biomedical applications. It keeps the most important 
merits of electrical impedance tomography (real-time imaging, low cost, portabil- 
ity). It is based on the simultaneous measurement of an electric current and of 
acoustic vibrations induced by ultrasound waves. Its intrinsic resolution depends 
on the size of the focal spot of the acoustic perturbation, and thus, it may provide 
high-resolution images. 

The core idea of impediography is to couple electric measurements to localized 
elastic perturbations. A body (a domain Q C R?) is electrically probed: one or 
several currents are imposed on the surface and the induced potentials are measured 
on the boundary. At the same time, a circular region of a few millimeters in 
the interior of is mechanically excited by ultrasonic waves, which dilate this 
region. The measurements are made as the focus of the ultrasounds scans the entire 
domain. Several sets of measurements can be obtained by varying amplitudes of the 
ultrasound waves and the applied currents. 

Within each disk of (small) volume, the conductivity is assumed to be constant 
per volume unit. At a point x € Q, within a disk D of volume Vp, the electrical 
conductivity y is defined in terms of a density p as y(x) = p(x)Vp. 

The ultrasonic waves induce a small elastic deformation of the disk D. If this 
deformation is isotropic, the material points of D occupy a volume ve in the 
perturbed configuration, which at first order is equal to 


- Ar 


r 


where r is the radius of the disk D and Ar is the variation of the radius due to the 
elastic perturbation. As Ar is proportional to the amplitude of the ultrasonic wave, 
one obtains a proportional change of the deformation. Using two different ultrasonic 
waves with different amplitudes but with the same spot, it is therefore easy to 
compute the ratio Ve / Vp. As a consequence, the perturbed electrical conductivity 
y? satisfies 


VxEQ, y?(x) = p(x)VP = y(x)n(x), 


where n(x) = es /Vp is a known function. One makes the following realistic 
assumptions: (1) the ultrasonic wave expands the zone it impacts and changes 
its conductivity, Vx € Q, n(x) > 1, and (2) the perturbation is not too small, 
n(x) —1 >> Vp. 
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Mathematical Model 


Let u be the voltage potential induced by a current g, in the absence of ultrasonic 
perturbations. It is given by 


V-(v(x)Vu) = 0inQ, 
(44) 


ye = g ondQ, 
dv 


with the convention that /, 9Q 4 = 0. One supposes that the conductivity y of the 
region close to the boundary of the domain is known, so that ultrasonic probing is 
limited to interior points. One denotes the region (open set) by (2). 

Let us be the voltage potential induced by a current g, in the presence 
of ultrasonic perturbations localized in a disk-shaped domain D := z+ 
6B of volume |D| = O (5?). The voltage potential us is a solution to 


V- (ys(x)Vus(x)) = 0 in Q, 
(45) 


ps = g ondQ, 
dv 


with the notation 


ys(x) = y(x) [1 + x(D) (x) (n(x) — 1D], 


where x(D) is the characteristic function of the domain D. 

As the zone deformed by the ultrasound wave is small, one can view it as a 
small-volume perturbation of the background conductivity y and seek an asymptotic 
expansion of the boundary values of us —u. The method of small-volume expansions 
shows that comparing us and u ondQ provides information about the conductivity. 
Indeed, one can prove that 


= (n(x) — 1)" 
[ -wedo = y (x) ————— nx) + 1 Vu-Vudx +o0(|D|) 


= veo vucp f C=O a4 o(Db. 


Note that because of assumption (2) at the end of the previous section, it follows 
that 


(n(x) — 1)° 


dx >C|D 
p n(x)+1 ee. 
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for some positive constant C. Therefore, one has 
v(z) |Vu(z)? = E@) + 0(1), (46) 


where the function €(z) is defined by 


_(fa@-y,\" 


By scanning the interior of the body with ultrasound waves, given an applied 
current g, one then obtains data from which one can compute the electrical energy 


E() = y@|Vu@/? 
in an interior subregion of Q. The new inverse problem is now to reconstruct y, 
knowing €. 
Substitution Algorithm 


The use of € leads one to transform (44), having two unknowns y and u with highly 
nonlinear dependency on y, into the following nonlinear PDE (the 0—Laplacian) 


Vx (<2) =0imnQ, 


Ive" 
u 
|Vul2 av =g 0n dQ. 


(48) 


It is worth emphasizing that € is a known function, constructed from the measured 
data (47). Consequently, all the parameters entering in (48) are known. Thus, the 
ill-posed inverse problem in electrical impedance tomography is converted into a 
less-complicated direct problem (48). 

The E-substitution algorithm, which will be explained below, uses two currents 
gi and gy. One chooses this pair of current patterns to have Vu, x Vun 4 
0 forall x € Q, where u;,i = 1,2, is the solution to (44). One refers to [66] 
and the references therein for an evidence of the possibility of such a choice. The 
E-substitution algorithm is based on an approximation of a linearized version of 


problem (48). 
Suppose that y is a small perturbation of conductivity profile yo : y = 
yo + dy. Let up and u = uo + du denote the potentials corresponding to 


yo and y with the same Neumann boundary data g. It is easily seen that 
du satisfies V- (yVdu) = —V-(dyVuo) in Q with the homogeneous Dirichlet 
boundary condition. Moreover, from 
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E = (yo + By) |V(uo + 6u)|* & yo| Vuol” + dy|Vuol” + 2voVuo- Vou, 


after neglecting the terms Sy Vuo - Vou and dy|Vdul’, it follows that 


y &% —, — Yo — 2¥9 ——— 
|Vuol|” |Vuol” 


The E-substitution algorithm is as follows. One starts from an initial guess for 
the conductivity y and solves the corresponding Dirichlet conductivity problem 


V- (yVuo) = 0 in, 
uo = w ondQ. 


The data y is the Dirichlet data measured as a response to the current g (say g = g1) 
in the absence of elastic deformation. The discrepancy between the data and the 
guessed solution is 


E 
€& := —, -}/. (49) 
|Vuol” 


One then introduces a corrector, du, computed as the solution to 


V: (yVéu) = —V«(e9Vuo) in Q, 


bu = 0ondQ, 
and updates the conductivity 
E —2yVébu-Vuo 
= = ee 
| Vol? 


One iteratively updates the conductivity, alternating directions of currents (i.e., with 
& = 82). 

Consider a disk-shaped domain Q, which contains three anomalies, an ellipse, 
an L-shaped domain, and a triangle. See Fig. 6. 

Figure 7 shows the result of the reconstruction when measurements with very 
accurate precision for two directions of currents are available. 

In the case of incomplete data, that is, if € is only known on a subset Q’ of the 
domain, one can follow an optimal control approach. See [39]. 


Bibliography and Open Questions 


Impediography was proposed in [8], and the substitution algorithm proposed there. 
An optimal control approach for solving the inverse problem in impediography has 


566 H. Ammari and H. Kang 


Fig. 6 Conductivity 
distribution 


{ {J 


Fig. 7 Reconstruction test. From /eft to right, the initial guess, the collected data € for two 
directions of currents, and the reconstructed conductivity 


been described in [39]. The inversion was considered as a minimization problem, 
and it was performed in two or three dimensions. 

As pointed out in [71], the success of impediography depends on the feasibility 
of focusing ultrasound waves at an arbitrary point inside the body. Such a focusing, 
however, is quite tricky to achieve in practice. See, for instance, [85]. A method 
to extract the measurements corresponding to well-focused beams from the data 
obtained with unfocused waves has been proposed in [71]. 

An interesting problem is to study the sensitivity of the inversion methods to 
limitations on the intensities of the applied voltages, as electrical safety regulations 
limit the amount of the total current that patients can sustain. Another interesting 
problem is to reconstruct anisotropic conductivity distributions and to see whether 
or not impediography allows one to remove the obstruction to unique identifiability 
of the conductivity by electrical impedance tomography. In electrical impedance 
tomography, it is known that any change of variables of the background conductor 
that leaves the boundary fixed gives rise to a new anisotropic conductivity with the 
same boundary measurements [68]. 
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6 Magneto-Acoustic Imaging 


In magneto-acoustic imaging, a probe signal such as an acoustic wave or an electric 
current (or voltage) is applied to a biological tissue placed in a magnetic field. The 
probe signal produces, by the Lorentz force, an induced signal that is a function 
of the local electrical conductivity of the biological tissue. If the probe signal is an 
acoustic wave, then the induced signal is an electric current and the Lorentz force 
causes a local current density. 

Induced boundary currents or pressure which are proportional to the local 
electrical conductivity can be measured to reconstruct the conductivity distribution 
with the spatial resolution of the ultrasound. The induced signal is detected, and an 
image of the local electrical conductivity of the specimen is generated based on the 
detected induced signal. The first method is referred as magneto-acousto-electrical 
tomography and the second one as magneto-acoustic tomography with magnetic 
induction. 


Magneto-Acousto-Electrical Tomography 


Physical Principles 

In magneto-acousto-electrical imaging, an acoustic wave is applied to a biological 
tissue placed in a magnetic field. The probe signal produces by the Lorentz force an 
electric current that is a function of the local electrical conductivity of the biological 
tissue [80]. The mathematical basis for this magneto-acoustic imaging approach is 
provided, and an efficient algorithm for solving the inverse problem is proposed 
which is quite similar to the one designed for impediography. 


Mathematical Model 
Denote by y(x) the unknown conductivity, and let the voltage potential v be the 
solution to the conductivity problem 


V-yVv=0inQ, 


v=gond0Q. ay) 


Suppose that the conductivity y is a known constant on a neighborhood of the 
boundary 0Q and let y, denote y|aq. 

In magneto-acoustic imaging, ultrasonic waves are focused on regions of small 
diameter inside a body placed on a static magnetic field. The oscillation of each 
small region results in frictional forces being applied to the ions, making them 
move. In the presence of a magnetic field, the ions experience a Lorentz force. This 
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gives rise to a localized current density within the medium. The current density is 
proportional to the local electrical conductivity [80]. In practice, the ultrasounds 
impact a spherical or ellipsoidal zone, of a few millimeters in diameter. The induced 
current density should thus be sensitive to conductivity variations at the millimeter 
scale, which is the precision required for breast cancer diagnostics. 

Let z € Q and D be a small impact zone around the point z. The created current 
by the Lorentz force density is given by 


I(x) = cx(D)(x)y (x)e, (51) 


for some constant c and a constant unit vector e, both of which are independent of 
z. With the induced current J,, the new voltage potential, denoted by u,, satisfies 


V- (yVuz + J.) =0inQ, 
uz = g ondQ. 


According to (51), the induced electrical potential w, := v — u, satisfies the 
conductivity equation: 
V-yvyVw,=cV-(y(D) ye) for x € Q, (52) 
w(x) = 0 for x € dQ. 


The inverse problem for the magneto-acousto-electrical imaging is to reconstruct 
the conductivity profile y from boundary measurements of Me Nag or equivalently 


Wel ag forze€ Q. 


Substitution Algorithm 
Since y is assumed to be constant in D and |D| is small, one obtains using Green’s 
identity 


dw, 
7 ya gdo ~ —c|D|V(yv)(2)-e. (53) 
aQ Vv 


The relation (53) shows that, by scanning the interior of the body with ultra- 
sound waves, cV(yv)(z)-e can be computed from the boundary measurements 
ons ag in Q. If one can rotate the subject, then cV(yv)(z) for any z in Q can be 
reconstructed. In practice, the constant c is not known. But, since yv and d(yv)/dv 
on the boundary of Q are known, one can recover c and yv from cV(yv) in a 
constructive way [11]. 


The new inverse problem is now to reconstruct the contrast profile y, knowing 


E(z) = y(z)v(z) (54) 


for a given boundary potential g, where v is the solution to (50). 


Expansion Methods 569 


In view of (54), v satisfies 


é : 
VS ve, (55) 


v=gond0Q. 


If one solves (55) for v, then (54) yields the conductivity contrast y. Note that to be 
able to solve (55), one needs to know the coefficient €(z) for all z, which amounts 
to scanning all the points z € Q by the ultrasonic beam. 

Observe that solving (55) is quite easy mathematically: if one puts w = Inv, then 
w is the solution to 


V-EVw = 0inQ, 

w = Ing on dQ, (56) 
as long as g > 0. Thus, if one solves (56) for w, then v := e” is the solution to 
(55). However, taking an exponent may amplify the error which already exists in 
the computed data €. In order to avoid this numerical instability, one solves (55) 
iteratively. To do so, one can adopt an iterative scheme similar to the one proposed 
in the previous section. 

Start with yo and let vo be the solution of 


V-yoVvo =0Oin Q, 


57 
vo = g ondQ. OD) 
According to (54), the updates, yo + dy and vo + dv, should satisfy 
& 
+ dy = ——_., 58 
" : vo + dv (8) 
where 
V-(vo + by) V (vo + bv) = On Q, 
6v=0ond0Q, 
or equivalently 
V-yoVov+ V-dyVvo = 0 in Q, (59) 
6v = 0 ondQ. 


One then linearizes (58) to have 


e z 3 
yo t+ by = (1 = *). (60) 


vo(1 + v/v) Vo Vo 
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Fig. 8 Reconstruction test. From Jeft to right, the conductivity distribution, the initial guess, and 
the reconstructed conductivity after three iterations 


Thus, 
E 
by ==——, = 6, pat YO (61) 


One then finds dv by solving 


V-ypVbv—V- (S25v) = V-8Vv in Q, 


0 (62) 
6v=O0 ond. 


Figure 8 shows the result of the reconstruction when very accurate measurements 
for two Dirichlet boundary conditions, g = g}, go, are available. 

In the case of incomplete data, that is, if € is only known on a subset w of the 
domain, one can follow an optimal control approach. See [11]. 


Magneto-Acoustic Imaging with Magnetic Induction 


Physical Principles 

In the magneto-acoustic tomography with magnetic induction, pulsed magnetic 
stimulation by the ultrasound beam is imposed on an object placed in a static 
magnetic field. The magnetic stimulation can be considered as an ideal pulsed 
distribution over time. The magnetically induced eddy current is then subject 
to a Lorentz force. This in turn creates a pressure wave that can be detected 
using an ultrasound hydrophone [80]. The magneto-acoustic tomography with 
magnetic induction uses this acoustic pressure wave to reconstruct the conductivity 
distribution of the sample as the focus of the ultrasound beam scans the entire 
domain. 
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Mathematical Model 

Let y be the conductivity distribution of the object as before. Denoting the constant 
magnetic field as Boy and the magnetically induced current density distribution as 
J.(x) with z indicating the location of the magnetic stimulation, the Lorentz force is 
given by 


J-(x) x Bod;=0 = cx(D)(x)y(x)e5;=0, 


where D is the impact zone which is a small neighborhood of z as before, and c is 
a constant independent of z and x. Then the wave equation governing the pressure 
distribution p, can be written as 


& p: 
or? 


—c?Ap,=0, xEQ, t€]0,TI, (63) 


for some final observation time 7’, where c, is the acoustic speed in Q. The pressure 
satisfies the Dirichlet boundary condition 


P2=90 ondQx)0, T[ (64) 
and the initial conditions 
Op: 
Pz|:=0 =O and eater —cV-(y(D)ye) inQ. (65) 
t= 


The inverse problem for the magneto-acoustic tomography with magnetic induc- 
tion is to determine the conductivity distribution y in Q from boundary measure- 
ments of ae on 0Qxj]0, T[ for all z € Q. Suppose that 7 is large enough so that 


diam(Q 
S jam(Q2) 
Cs 


T : (66) 


which says that the observation time is long enough for the wave initiated at z to 
reach the boundary 0Q. 


Reconstruction Algorithm 

The algorithms for the magneto-acoustic tomography with magnetic induction 
available in the literature are limited to unbounded media. They use the spherical 
Radon transform inversion. However, the pressure field is significantly affected by 
the acoustic boundary conditions at the tissue—air interface, where the pressure must 
vanish. Thus, one cannot base magneto-acoustic imaging on pressure measurements 
made over a free surface. Instead, one can use the following algorithm. 
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Let w satisfy 


ew yg : 
ga cs Aw 0 in Qx]0, TI, (67) 
with the final conditions 
hao = in Q. 
wher va O in (68) 


Since y is constant on D, one can prove that the following identity holds: 


i [4 sale, t)w(x,t) do(x) dt = Sv@) fe: Vw(x.0)de. (69) 


Suppose that d = 3. For y € R*\Q, define the probe function 


AG +T- — 


wy (x,t) = er 


in Qx]0, TI, (70) 


where t := 2 2 =| The function wy is a Green’s function corresponding to retarded 


potentials. Choosing wy as a test function in (69) yields the new identity 


re : Op: 
[poe Ww, (x, 0)dx I [, By er Ewe) do(x) dt. Ci) 
D a? 


cy(z) = 


The quantity /,,e-Vw,(x,0)dx can be explicitly computed. In particular, if the 
source point y is such that z — y is parallel to e and D is a sphere of radius r (and 
center z), then 


cy(z) = f° a Oi BP t)wy (x, t) do(x) dt, (72) 


Ae-yP  ale—-yFF 


provided that y is constant on D. But since r is sufficiently small, one obtains 


_ yj2 pT 
cy(2) © ee | / ee Aye ae de. (73) 
0 Jag ov 


Formula (73) can be used to effectively compute the conductivity contrast in Q 
with a resolution of order the size of the ultrasound beam. It is worth emphasizing 
that unlike magneto-acousto-electrical imaging, in magneto-acoustic tomography 
with magnetic induction, it suffices to excite the local spot at z in order to obtain the 
value cy(z), as clearly shown by (73). 
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Bibliography and Open Questions 


The feasibility of magneto-acoustic imaging has been demonstrated in [54, 75, 76]. 
The mathematical and numerical modeling described in this section is from [11]. As 
it will be shown in Sect. 8, the approach for the magneto-acoustic tomography with 
magnetic induction can be used in photo-acoustic imaging. 

It would be interesting to prove the convergence of the proposed iterative scheme 
for magneto-acousto-electrical tomography. Another important problem is to design 
an efficient inversion algorithm for magneto-acoustic tomography with magnetic 
induction when the acoustic speed fluctuates randomly. 


7 Magnetic Resonance Elastography 
Physical Principles 


Extensive work has been carried out in the past decade to image, by inducing 
motion, the elastic properties of human soft tissues. This wide application field, 
called elasticity imaging or elastography, is based on the initial idea that shear 
elasticity can be correlated with the pathological state of tissues. Several techniques 
arose according to the type of mechanical excitation chosen (static compression, 
monochromatic, or transient vibration) and the way these excitations are generated 
(externally or internally). Different imaging modalities can be used to estimate the 
resulting tissue displacements. 

Magnetic resonance elastography (MRE) is a new way of realizing the idea of 
elastography. It can directly visualize and quantitatively measure the displacement 
field in tissues subject to harmonic mechanical excitation at low frequencies. A 
phase-contrast magnetic resonance imaging technique is used to spatially map and 
measure the complete three-dimensional displacement patterns. From this data, 
local quantitative values of shear modulus can be calculated, and images that depict 
tissue elasticity or stiffness can be generated. The inverse problem for magnetic 
resonance elastography is to determine the shape and the elastic parameters of 
an elastic anomaly from internal measurements of the displacement field. In most 
cases, the most significant elastic parameter is the stiffness coefficient. 

In biological media, the compression modulus is four to six orders higher than the 
shear modulus. One can prove that, as the compression modulus goes to +00, the 
Lamé system converges to the modified Stokes system. By reducing the elasticity 
system to a modified Stokes system, one removes the compression modulus from 
consideration. 


Mathematical Model 


Consider the modified Stokes system, i.e., the problem of determining v and g ina 
domain Q from the conditions: 
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V-v=0, (74) 


Problem (74) governs elastic wave propagation in nearly incompressible homoge- 
neous media. 

Let (G; aM j= be the Dirichlet Green function for the operator in (74), i.e., for 
ye, 


OF; (x — 
(A, + x’) Gii(x, y) _ “ee = 816) (x) inQ, 
da : 
> —Gilx.y) =0 inQ, (75) 
l=1 Ox) 


Gils, y) =0 ondQ. 


Denote by (e;,..., ey) an orthonormal basis of R?. Let d(&) := (1/d) °; ex 
and Voge for p,q = 1,...,d, be the solution to 


UAV g + VP =0 inR4\B, 
pV pg +VP=0 inB, 


Vngl- — Vpgl+ = 9 on OB, 
av ov 
BN + 7@— )|_ —[ BN _ Pd —0 OB, (76) 
G + ae) G + Bay + on 


V-¥pq =0 inR?, 
Vg (E) > Epeg — Spgd (E) as |&| > 00, 
P(E) +0. as |&| > +00. 


Here, dv/dN = (Vv + (Vv)*)-N and (Vv)* denotes the transpose of the matrix 
Vv. 
Define the viscous moment tensor (Vijpq)i,j,p,q=1 


Ving = (fe —B) [ Ving «(V(Ee;) + V(Eie))*) dé. (77) 


Consider an elastic anomaly D inside a nearly compressible medium (2. The 
anomaly D has a shear modulus jx different from that of Q, 4. The displacement 
field u solves the following transmission problem for the modified Stokes problem: 
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(uA+o°)u+Vp=0 inQ\D, 
(fA +o°)u+Vp=0 in D, 


u|_ =u, on 0D, 
du _ ou 
(Ple- PLIN+ HGR] —figR| = 0 on 0D, (78) 
V-u=0 inQ, 
u=g ondQ, 


[o=e 
Q 


where g € L?(4Q) satisfies the compatibility condition / g-N=0. 


IQ 
The inverse problem consists of reconstructing jz and the shape of the inclusion 
D from internal measurements of u. 


Asymptotic Analysis of Displacement Fields 


Let (U,q) denote the background solution to the modified Stokes system in the 
absence of any anomalies, that is, the solution to 


(uA + w*)U+Vq =0 inQ, 
V-U=0 inQ, 
U=g ondQ, (79) 


[a-o 
2 


The following asymptotic expansions hold. 


Theorem 8 (Expansions of the displacement field). Suppose that D = 6B + z 
and let u be the solution of (78), where 0 < ju # fh < +00. 


(i) The following inner expansion holds: 
d 


u(x) © UG) +8 > %U@p%py ( 


p.q=l 


x= Z 


; ) for x near z, (80) 


where ¥V pq is defined by (76). 
(ii) Let (Vijpq) be the viscous moment tensor defined by (77). The following outer 
expansion holds uniformly for x € 0Q: 
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d 
(u—U)(x) = 84 |S) ej Gei (x, 2)OgU() pViig | « (81) 


ij,p.ql=1 


where Vijpg is given by (77), and the Green function (Gi)4,—, is defined by (75) 
with k> = w*/, being the shear modulus of the background medium. 


The notion of a viscous moment tensor extends the notion of a polarization tensor 
to quasi-incompressible elasticity. The viscous moment tensor, V, characterizes all 
the information about the elastic anomaly that can be learned from the leading- 
order term of the outer expansion (81). It can be explicitly computed for disks and 
ellipses in the plane and balls and ellipsoids in three-dimensional space. If B is a 
two-dimensional disk, then 


Vv =4)3[pA—) p, 
jit 


where P = (Pijpa) is the orthogonal projection from the space of symmetric 
matrices onto the space of symmetric matrices of trace zero, i.e., 


1 1 
Pring = 2 (5:p5jq ae 5:95 jp) = oui Sa: 


If B is an ellipse of the form 


2 2. 
$+ =1, g= te (82) 


then the viscous moment tensor for B is given by 


Sth = 
Vita = Vo222 = —Vi122 = —V2211 = | BI ae H) a 
eae = i —p)m 
Vi2i2 = Vari2 = Vier = Vai2i = |B| eee (83) 


w+ e+ (A — ym?’ 
the remaining terms are zero, 


where m = (a —b)/(a +b). 
If B is a ball in three dimensions, the viscous moment tensor associated with B 
and an arbitrary j2 is given by 
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— 20n/B| H-Bw  y _ _10p|B) h—-w 


Viiii = » Vay z i # fj), 
3 mts 3 aan 7 YP 
Ue : , (84) 
Vigiy = Viggi = S| Bl = ,@ ; 
jij ii LL| a+ 3a Gi # j) 


the remaining terms are zero. 


Theorem 9 (Properties of the viscous moment tensor). For 0 < fi # mh < 


+oo, lettV = (Vizog)$ » oi be the viscous moment tensor associated with the 


bounded domain B in R@ and the pair of shear modulus (jt, 1). Then 


(i) Fori,j,p,q =1,...,d, 
Vijpg = Viivg> Viipa = Vijqp» Viipg = Vogij- (85) 
(ii) One has 


Yo Viinp =0 foralli,j and S Viipg =0 forall p,q. 


Pp i 


or equivalently, V = PVP. 

(iii) The tensor V is positive (negative, resp.) definite on the space of symmetric 
matrices of trace zero if [ft > [L (jL < [L, resp.). 

(iv) The tensor (1/(2k2)) V satisfies the following bounds: 


i3|(4-1) (a-n£+), (36) 
be bh 2 


i..\ 1 ( m “a= ») 
iv) S-—  a (87) 
(a, ) || (E-1) be 2 


where for C = (Gijpa)s Tr(C) = pa Cyiy. 


ij=l 


= 
——~ 
i) 
B/- 

= 
Ne 

lA 


A 


Note that the viscous moment tensor, V, is a four tensor and can be regarded, 
because of its symmetry, as a linear transformation on the space of symmetric 
matrices. Note also that, in view of Theorem 2, the right-hand sides of (86) and (87) 
are exactly in the two-dimensional case (d = 2) the Hashin—Shtrikman bounds (9) 
for the polarization tensor associated with the same domain B and the conductivity 
contrast k = ji/u. 
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Numerical Methods 


Let u be the solution to the modified Stokes system (78). The inverse problem in the 
magnetic resonance elastography is to reconstruct the shape and the shear modulus 
of the anomaly D from internal measurements of u. 

Based on the inner asymptotic expansion (80) of du (:= u — U) of the 
perturbations in the displacement field that are due to the presence of the anomaly, 
a reconstruction method of binary level set type can be designed. 

The first step for the reconstruction procedure is to locate the anomaly. This can 
be done using the outer expansion of du, i.e., an expansion far away from the elastic 
anomaly. 

Suppose that z is reconstructed. Since the representation D = z+ 6B is not 
unique, one can fix 5. One uses a binary level set representation f of the scaled 
domain B: 


F0)= | tT RAE (88) 
Let 
nn) +)-M(E)- 


and let 6 be a regularization parameter. Then the second step is to fix a window W 
(containing z) and solve the following constrained minimization problem 


2 


1 ; _ 
min L(f, jt) =5 |] 8u(x) —8 D> 8,UQ) p¥py (=) + VU(2)(x — 2) 
bof p.q=l L2(W) 
+6 |Vh(x)| dx, (90) 
Ww 


subject to (76). Here, fj, |VA| dx is the total variation of the shear modulus and 
|Vh| is understood as a measure: 


[wn = sun} | hV-vdx, v€Cj(W) and |v| <linW?. 
Ww W 


This regularization indirectly controls both the length of the level curves and the 
jumps in the coefficients. 
The local character of the method is due to the decay of 


d 
§ > aU@p¥pq (=) —~VU@(- -2) 


p.q=l 
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Fig. 9 Reconstruction using the data on the whole domain on the /eft, a zoom on the anomaly 


in the middle, and on the right the reconstruction limited on the subregion defined by the boxed 
region on the left 


away from z. This is one of the main features of the method. In the presence of noise, 
because of a trade-off between accuracy and stability, one has to choose carefully 
the size of W. As it has been shown in [13], the size of W should not be too small in 
order to preserve some stability and not too big so that one can gain some accuracy. 
See Fig. 9. 

The minimization problem (90) corresponds to a minimization with respect to ft 
followed by a step of minimization with respect to f. The minimization steps are 
over the set of (4 and f and can be performed using a gradient-based method with a 
line search. Of importance are the optimal bounds satisfied by the viscous moment 
tensor V. One should check at each step whether the bounds (86) and (87) on V are 
satisfied or not. In the case where they are not, one has to restate the value of jL. 
Another way to deal with (86) and (87) is to introduce them into the minimization 
problem (90) as a constraint. Set a = Tr(V) and B = Tr(V~') and suppose for 
simplicity that ij > jy. Then, (86) and (87) can be rewritten (when d = 3) as 
follows: 


7 2 
r2G=y (3+ “PY (DI 
2U( — 1) 
3u + 2p 


(91) 
[jpl= po. 


Bibliography and Open Questions 


Magnetic resonance elastography was first proposed in [81]. The results provided 
on this technique are from [14]. Theorem 8 and the results on the viscous moment 
tensor in Theorem 9 are from [15]. 
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In general, the elastic parameters of biological tissues show anisotropic proper- 
ties, that is, the local value of elasticity is different in the different spatial directions 
[91] and also viscous properties. It would be very interesting to extend the algorithm 
described in this section for detecting the shape of an elastic anomaly and the 
viscosity and the anisotropy in its shear modulus. The study of the dependence of 
the shear modulus as a function of the frequency is also important [90]. 


8 Photo-Acoustic Imaging of Small Absorbers 
Physical Principles 


In photo-acoustic imaging, optical energy absorption causes thermoelastic expan- 
sion of the tissue, which in turn leads to propagation of a pressure wave. This signal 
is measured by transducers distributed on the boundary of the object, which is in turn 
used for imaging optical properties of the object. The significance of photo-acoustic 
imaging is to provide images of optical contrasts (based on the optical absorption) 
with the resolution of ultrasound. 

In pure optical imaging, optical scattering in soft tissues degrades spatial 
resolution significantly with depth. As for electrical impedance tomography, even 
though pure optical imaging is very sensitive to optical absorption, it can only 
provide a spatial resolution of the order of 1 cm at centimeter depths. As discussed 
before, pure conventional ultrasound imaging is based on the detection of the 
mechanical properties (acoustic impedance) in biological soft tissues. It can provide 
good spatial resolution because of its millimetric wavelength and weak scattering at 
megahertz frequencies. 

If the medium is acoustically homogeneous and has the same acoustic properties 
as the free space, then the boundary of the object plays no role and the optical 
properties of the medium can be extracted from the measurements of the pressure 
wave by inverting a spherical Radon transform. 

In the more realistic situation, where a boundary condition has to be imposed 
on the pressure field, such an inversion formula does not hold. Using asymptotic 
analysis, one can develop an efficient approach for reconstructing absorbing regions 
and absorbing energy density inside a bounded domain from boundary data. One 
can also reconstruct the optical absorption coefficient. In general, it is not possible 
to infer physiological parameters from the absorbing energy density. It is the optical 
absorption coefficient distribution that directly correlates with tissue structural and 
functional information such as blood oxygenation. 


Mathematical Model 


Let D;,J = 1,...,m, be m absorbing domains inside the nonabsorbing 
background-bounded medium Q C R?%,d = 2 or 3. In an acoustically 
homogeneous medium, the photo-acoustic effect is described by the following 
equation: 
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2: 
TP x,t) PAPO.) = E(x.) xeEQ, teER, (92) 


where c, is the acoustic speed in Q2, y the dimensionless Griineisen coefficient in 
Q, and H(x,t) a heat source function (absorbed energy per unit time per unit 
volume). 

Assuming the stress-confinement condition, the source term can be modeled 
as yH(x,t) = 6(t)A(x), where the absorbed optical energy density times the 
Griineisen coefficient A = )\/_, Ai x(D_) and A; are constants. Under this assump- 
tion, the pressure in an acoustically homogeneous medium obeys the following wave 


equation: 


ap 


az 9) —c?Ap(x,t)=0, xEQ, t€]0,TI, 


for some final observation time T. The pressure satisfies the Dirichlet boundary 
condition 


p=09 ondQx)0, T| 


and the initial conditions 


m Op 
Plimo = 2 XCD i)As and Fh co = 


Suppose that T satisfies (66). The inverse problem in photo-acoustic imaging is 
to determine the supports of nonzero optical absorption (D;,/ = 1,...,m) in Q 
and A(x) from boundary measurements of op on 0Qx]0, T[. 


Reconstruction Algorithms 
Analogously to (71), the following identity holds: 
1 m T dp 
a d A : d:wy(x,0; t)dx = hs ay h Owy(ats t)do(x)dt, (93) 


where the probe function w, is given by (70). 


Determination of Location 
Suppose for simplicity that there is only one absorbing object (m = 1) which is 
denoted by D (= z+ 6B). Identity (93) shows that the imaging functional 
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Fig. 10 Real configuration of the medium on the /eft — there are seven optical anomalies of 
various size and absorption. Reconstructed configuration on the right — anomalies 6 and 7 are 
reconstructed as a single anomaly 


T Op 
Wt, y):= [ a By wy Or, t;t)do(x) dt (94) 


is nonzero only on the interval |t,, t-[, where t, = dist(y, D)/c, is the first t for 
which the sphere of center y and radius t hits D and t; is the last t for which such 
sphere hits D. This gives a simple way to detect the location (by changing the source 
point y and taking intersection of spheres). The functional W(t, y) can be used to 
probe the medium as a function of t and y. For fixed y, it is a one-dimensional 
function and is related to time reversal in the sense that it is a convolution with a 
reversed wave. 

A result of numerical simulation to validate the location search algorithm is given 
in Fig. 10. 


Estimation of Absorbing Energy 


Consider first the three-dimensional case. If D is a sphere with A(x) = Ax(D), 
then one has 


8A & o,|z— vi f 


T ap 
i} i (x, t)wy (x,t; t) do(x) dt | dt, (95) 
0 Jaa Ov 


which gives an approximation of 67 A. 
In two dimensions, one should rather consider the probe wave given by 


Gen =8(r+e- =A), (96) 


where @ is a unit vector and t is a parameter satisfying 


Expansion Methods 583 


One can still use the function 


T ap 
T vf / (x, t)we(x, t; t) do(x) dt 
0 Jag Ov 


to probe the medium as a function of t. This quantity is nonzero on the interval 
]ta, Te[, where t, and t, are defined such that planes (x,@) = ct for t = tT, and 
Tt. hit D. Changing the direction @ and intersecting stripes gives an efficient way to 
reconstruct the anomalies. 

By exactly the same arguments as in three dimensions, one can show that 


Cs Te T Op 
saw i / —(x, t)wo (x,t; t) do(x) dt} dt. (97) 
4 Ta 0 AQ dv 


The above formula can be used to estimate 5A. 
In the case when there are m inclusions, one first computes for each / the quantity 


O; best = argmax (min |(z; — z, 0) ) 
de[0.x] \J#l 


and then, since along the direction 4 pest, the inclusion D; is well separated from all 
the other inclusions, one can use formula (97) to estimate its 6A). 


Reconstruction of the Absorption Coefficient 

The density A(x) is related to the optical absorption coefficient distribution 
Ma(x) = Uax(D) by the equation A(x) = pag(x)®(x), where ® is the fluence 
rate. The function ® depends on the distribution of scattering and absorption within 
(2, as well as the light sources. Based on the diffusion approximation to the transport 
equation, ©® satisfies 


iw 1 1 
(2 + matey - 39 


: V = inQ, 
3 eee CS ) 269 O in (98) 


with the boundary condition 
——=g ond. (99) 
v 


Here, g denotes the light source, w a given frequency, c the speed of light, and jus 
the scattering coefficient. The diffusion approximation holds when pls >> [a- 
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Suppose that d = 3 and jus is known a priori. Define Bp by 


io 1 1 


(<- : <9) O(x) =0 inQ, 


subject to the boundary condition 


1 a® 
— =¢ on 0Q. 
Ms Ov 


Introduce Ng to be the Newton potential given by 


Np (&) := [ T(é —y) dy, 


where [ := —1/(42|x|) is a fundamental solution of the Laplacian in three 
dimensions. 

Let w := 87 14®(z) = 87 A. As shown before, can be reconstructed from a. 
To extract 57/1, from a, one uses the following theorem: 


Theorem 10 (Fluence rate perturbations). Jf B is the unit sphere, then the 
following expansion holds: 


(® — ©p)(z) © 3aps(z)Np (0), (100) 


from which it follows that the (normalized) absorption coefficient can be approxi- 
mated by 


2 a 


ua, x _ (101) 
i 3aps(z)Np(0) + Bo(z) 


Separating 6 from {4g requires boundary measurements of ® on dQ. One can use 
i g(® — Bo) do & pla 5(2)|D| (102) 
aQ 


to separately recover 6 from [Ug. 

In the case where jus is unknown, an algorithm to extract the absorption coeffi- 
cient (4, from absorbed energies obtained at multiple wavelengths was developed in 
[10]. It assumes that the wavelength dependence of the scattering and absorption 
coefficients are known. In biological tissues, the wavelength dependence of the 
scattering often approximates to a power law. 
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Bibliography and Open Questions 


Basic physical principles of the photo-acoustic effect have been described, for 
instance, in [49,98]. The results of this section are from [9, 10]. The location search 
algorithm described in this section can be extended to the case with limited-view 
measurements. The half-space problem has been considered [96]. In free space, one 
refers to [1, 2, 4,55, 56, 70, 84] for uniqueness of the reconstruction and inversion 
procedures based on the spherical Radon transform. Reconstruction methods with 
incomplete data have been developed in [97]. Sensitivity analysis of a photo- 
acoustic wave to the presence of small absorbing objects has been provided in [49]. 

In connection with photo-acoustic imaging, it is worth mentioning the multi- 
physics imaging technique proposed in [52], which combines electrical impedance 
tomography with acoustic tomography. This method makes use of the fact that the 
absorbed electrical energy causes thermoelastic expansion of the tissue, which leads 
to propagation of a pressure wave. With the notation of Sect. 5, the induced signal 
is measured on the boundary of the object and can be used for calculating the 
absorbed electrical energy, € = y|Vul?, inside the body, from which the electrical 
conductivity y can be reconstructed using, for instance, the substitution algorithm. 

As for magneto-acoustic imaging with magnetic induction, it would be very 
interesting to design a robust inversion algorithm when the acoustic speed fluctuates 
randomly. 


9 Conclusion 


In this chapter, applications of asymptotic analysis in emerging medical imaging are 
outlined. This method leads to very effective and robust reconstruction algorithms 
in many imaging problems. Of particular interest are emerging multi-physics or 
hybrid-imaging approaches. These approaches allow one to overcome the severe 
ill-posedness character of image reconstruction. It would be very interesting to ana- 
lytically investigate their robustness, with respect to incomplete data, measurement, 
and medium noises. Another important problem is to take into account the effect of 
anisotropy, dissipation, or attenuation in biological tissues. 
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Abstract 

The topic of this chapter is devoted to shape identification problems, i.e., 
problems where the shape of an object has to be determined from indirect 
measurements. In contrast to iterative methods where a sequence of forward 
problems has to be computed the sampling methods avoid the (usually expansive) 
computation of the forward problems. Instead, a class of test objects (e.g., 
points) are chosen and a binary criterium is constructed which depends on the 
measured data only, and which decides whether this test object is inside or 
outside of the searched for domain. In this chapter, the factorization method is 
explained for the impedance tomography problem with insulating or conducting 
inclusions, for scattering theory for time harmonic acoustic plane waves in the 
presence of a perfectly sound—soft obstacle, and for electromagnetic scattering by 
an inhomogeneous conducting medium. Brief descriptions of related sampling 
methods, such as the linear sampling method, MUSIC, the singular sources 
method, and the probe method complement this chapter. 


1 Introduction 


The topic of this chapter is devoted to shape identification problems, 1.e., problems 
where the shape of an object has to be determined from indirect measurements. 
Such a situation typically occurs in problems of tomography, in particular electrical 
impedance tomography or optical tomography. For example, a current through a 
homogeneous object will in general induce a different potential than the same 
current through the same object containing an enclosed cavity. In impedance 
tomography, the task is to determine the shape of the cavity from measurements 
of the potential on the boundary of the object. For survey articles on this subject, we 
refer to [18,54], and chapter » Electrical Impedance Tomography in this volume. 

As a second of these fields, we mention inverse scattering problems where one 
wants to detect — and identify — unknown objects through the use of acoustic, 
electromagnetic, or elastic waves. Similar to the above, one of the important 
problems in inverse scattering theory is to determine the shape of the scattering 
obstacle from field measurements. Applications of inverse scattering problems occur 
in such diverse areas as medical imaging, material science, nondestructive testing, 
radar, remote sensing, or seismic exploration. A survey on the state of the art of the 
mathematical theory and numerical approaches for solving inverse time-harmonic 
scattering problems until 1998 can be found in the standard monograph [36] (see 
also chapter » Inverse Scattering or [83] for an introduction and survey on inverse 
scattering problems). 

Shape identification problems are intrinsically nonlinear, i.e., the measured 
quantities do not depend linearly on the shape. Even the notion of linearity does 
not make sense since, in general, the set of admissible shapes does not carry a linear 
structure. Traditional (and still very successful) approaches describe the objects by 
appropriate parameterizations and compute the parameters by iterative schemes as, 
e.g., Newton-type methods. Newton-type methods are attractive because of their 
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fast convergence, although they require a good initial guess to converge. Still, 
these methods are widely used — partly because techniques from shape optimization 
theory can be used to characterize the required first- or second-order derivatives. We 
refer to [84, 89] for general references and to [57,58, 65] for applications in inverse 
scattering theory. 

While classical iterative algorithms use explicit parameterizations of the object, 
new shape optimization methods have been developed since around 1995 which 
completely avoid the use of parameterizations and replace the classical Fréchet 
derivative by a geometrically motivated topological derivative, see, e.g., [50] 
for the application of these methods in the inverse scattering context. Yet these 
methods have the shortcoming that they are not able to change the number of 
connectivity components during the algorithm. This has led to the development 
of level set methods which are based on implicit representations of the unknown 
object involving an “evolution parameter” t. We refer to [25] or chapter » Level Set 
Methods for Structural Inversion and Image Reconstruction for recent surveys. 

While very successful in many cases, iterative methods for shape identification 
problems — may they use classical tools as the Fréchet derivative or more recent 
techniques such as domain derivatives, level curves, or topological derivatives — are 
computationally very expensive since they require the solution of a direct problem 
in every step. Furthermore, for many important cases, the convergence theory is 
still missing. This is due to the fact that these problems are not only nonlinear but 
also because their linearizations are improperly posed. Although there exist many 
results on the convergence of (regularized) iterative methods for solving nonlinear 
improperly posed problems (see, e.g., [39, 64] or chapter » Iterative Solution 
Methods), the assumptions for convergence are not met in the applications to shape 
identification problems. (Or, at least, it is unknown whether these assumptions are 
fulfilled or not.) 

These difficulties and disadvantages of iterative schemes gave rise to the devel- 
opment of different classes of non-iterative methods which avoid the solution of a 
sequence of direct problems. We briefly mention decomposition methods (according 
to the notion of [37]) which consist of an analytic continuation step (which is linear 
but highly improperly posed) and a nonlinear step of finding the boundary of the 
unknown domain by forcing the boundary condition to hold. We refer to section 
“Decomposition Methods” in chapter » Inverse Scattering. 

This chapter will focus on a different class of non-iterative methods, the so-called 
sampling methods. The common idea of these methods is the construction of criteria 
on the known data to decide whether a given test object (a point or a curve or a 
set) is inside or outside the unknown domain D. Then, a grid of “sampling” points 
is chosen in a region that is known to contain the unknown domain D, in order 
to compute the (approximate) characteristic function of D. The different kinds of 
sampling methods differ in the way of defining the criteria and in the type of test 
objects. 

One of the first methods which falls into this class has been developed by 
David Colton and one of the authors (A. K.) in 1996 [35], now known as the 
linear sampling method. Its origin goes back to the dual space method developed 
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between 1985 and 1990 (see, e.g., [36]). The numerical implementation of the linear 
sampling method is extremely simple and fast because sampling is done by points 
z only. For every sampling point z, one has to compute the field of a point source 
in z with respect to the background medium (essentially, one has to compute the 
fundamental solution of the underlying differential operator; if the background is 
constant, the response is given analytically) and evaluate a series, i.e., a finite sum 
in practice. 

A problem with the linear sampling method from the mathematical point of view 
is that the computable criterion is only a sufficient condition which is, in general, not 
necessary. The factorization method overcomes this drawback and yields a criterion 
for z which is both necessary and sufficient. Therefore, this method succeeds to 
provide a simple formula for the characteristic function of D which can easily be 
used for numerical computations. 

The factorization method consists of three components. First, a “measurement 
operator” M is factorized in three factors of the form 


M = AGA*, (1) 


where A* is the dual operator of A with respect to the L? topology. Second, the 
range of A is characterized by the obstacle D, and vice versa. Third, if the operator 
G satisfies a certain coercivity condition, then the range of A can be determined 
by the given operator VM. This requires some functional analytic results on range 
identities which we have collected in an appendix. 

Combining these three steps yields an explicit characterization of the unknown 
obstacle D by the measurement operator M. 

The outline of this chapter is as follows. First, in Sect.2, we present the fac- 
torization method for two different settings in the impedance tomography context. 
In the very first setting, we deal with insulating inclusions, and this allows for 
a very elementary presentation of the method. Afterwards, in Sect.3, we turn to 
applications from inverse acoustic and (full 3D) electromagnetic scattering. Finally, 
we give a brief overview of other sampling type methods in Sect. 4, including the 
original linear sampling method and MUSIC-type methods. 


2 The Factorization Method in Impedance Tomography 


We start with the impedance tomography problem. Consider an object that fills a 
simply connected domain 2 C R"” with Lipschitz continuous boundary, where 
n = 2 orn = 3, respectively. We assume that the object is a homogeneous and 
isotropic conductor, except for a finite number m of so-called inclusions, given by 
domains D; C Q,i = 1,...,m, with Lipschitz continuous boundaries 0D;. We 
assume that these domains are well separated, i.e., D,;AD j = Owheni F j, 
and that the complement of the closure D of D = Ur, D; is connected. In 
impedance tomography, currents are imposed through the boundary of the object, 
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and the resulting boundary potentials are measured. Linear independent boundary 
currents yield independent pieces of information, which can be used as input data 
to determine the unknown shapes and positions of the inclusions. 

In practice, at least in most medical applications, the boundary currents have 
a frequency in the kHz range (5—500kHz), and the dc approximation with a 
positive real conductivity o (or possibly a positive definite tensor) serves as a 
suitable physical model. Without loss of generality, we can always assume that the 
homogeneous conductivity of the object equals 0 = 1, whereas o # 1 within the 
inclusions. 

Below we will consider two specific scenarios. In the first one, we assume that 
the inclusions are insulating, formally corresponding to the case where o = 0. Our 
analysis of the factorization method for the corresponding inverse problem will be 
somewhat nonstandard; in particular, we employ a factorization in only two factors 
instead of three as in (1), but this allows for a most elementary treatment of the 
method. 

Subsequently, we show how to deal with conducting obstacles with a conductiv- 
ity tensor o. Of particular interest is the setting where the object under consideration 
can be modeled as a half space; examples of this sort arise in geophysics, cf. [78], 
and in medicine, e.g., when a planar device is used for mammography examinations, 
cf. [92]. Another interesting application for the half-space problem has recently been 
considered in [17]. We therefore briefly describe the differences that arise in this 
context (mainly in the theoretical justification of the method). 

We conclude our case studies with a setting where the inclusion degenerates to 
a crack, i.e., an m — 1 dimensional smooth manifold within Q. This application 
requires some care in the appropriate implementation of the factorization method. 


Impedance Tomography in the Presence of Insulating Inclusions 


To begin with, we take up the case where Q is a bounded domain, and the domains 
D; C Q, i = 1,...,m correspond to insulating inclusions. Within the dc model, 
the potential uo induced by a boundary current f is given by 


= 0 
Auy=0 inQ\D, =u =0 ondD, 
dv 2 
' (2) 


—uy= f ondQ, [was =o. 
ov an 


where the normal vectors v on dQ and dD are pointing into the exterior of Q and 
D, respectively. In order to make the forward problem (2) well posed, we restrict 
f to be square integrable with vanishing mean on 02. The corresponding set of 
admissible boundary currents is 


L.00)= if € L7(dQ): i: fds = of. (3) 
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Under these assumptions, problem (2) has a unique (weak) solution 
uy € H3(Q\D) =iue H'(Q\D): i uds = of 
an 


The last condition in (2) normalizes this boundary potential to have vanishing mean; 
without this condition, the solution would only be unique up to additive constants, 
reflecting the fact that only the voltage, i.e., the difference between the potential at 
two different points, is a well-defined physical quantity. 

Therefore, the direct problem is to determine the field uw) when f and D are 
given. 

The quantity that is measured in impedance tomography is the trace go = uolag, 
i.e., the boundary potential. The corresponding measurement operator 


_{ 13,02) > 13,(0Q), 


Ao 
f+ 80 = Uolaa, 


(4) 


ie., the so-called Neumann-Dirichlet operator, is usually referred to as absolute 
data in impedance tomography. 

The inverse problem is to determine the shape of D from the measurement 
operator Ao. 

For the factorization method, we employ relative data, that is, the difference 
between the above Neumann-Dirichlet operator and the corresponding one for a 
completely homogeneous object in Q. To be precise, let uy be the reference solution 
for the homogeneous object, given the same boundary current f € L3(0Q), 


0 
Au; =0 inQ, —uy,= f ondaQ, / u, ds = 0, (5) 
ov dQ 


and denote by Aq: f +> gy = uy|aq the Neumann-Dirichlet map associated with 
(5). It is the relative data M = Ag — Aj, that later enters in (1) to lay the grounds 
for the setting of the factorization method. 

We refer to chapter > Electrical Impedance Tomography for a more elaborate 
treatment of the impedance tomography problem, but we will see below that Ap— Az 
is a bounded and positive self-adjoint operator. We also do not discuss practical 
issues such as electrode models that should be incorporated into a realistic problem 
setting. For the same reason, we do not comment on how to obtain relative data 
in practice; the generation of accurate reference data is indeed a difficult subject, 
and some work-arounds have therefore been suggested for this purpose. (We like 
to highlight one recent approach from [56], where different frequencies are used 
in the experimental setup to obtain relative data. This approach, however, leads to a 
different variant of the factorization method than the one that is described here.) Our 
specification of the impedance tomography problem is thus a purely mathematical 
one, although it can be shown to be a pretty reasonable approximation of the real 
case, cf., e.g., [60,77]. 
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Before we continue, we pause to comment on the nature of the relative data 
introduced above. Any function / in the range (Ap — Ay) of Ag— Ay corresponds 
to a suitable input current f € La (8a), such that h is the trace of w = uo — ut : 
Q\D — R, where uo and uy are the solutions of (2) and (5), respectively. As uo and 
uy are both harmonic in Q\D, the same holds true for w; on top of that, like up and 
uz, w has finite H! norm on Q\D, as well as vanishing mean on 0Q. Moreover, w 
has homogeneous Neumann boundary conditions on 0Q2, as uo and uy both satisfy 
the same Neumann boundary condition. And finally, on 0D;,i = 1,...,m, we have 


/ t yie= i de = 
aD; dv" = aD; av = 


by virtue of Green’s formula. Accordingly, the range of Ag — Ax consists of traces 
of potentials w from 


oe, a 
W= Jw € HS(Q\D) : Aw =0, w= 00n 09, 
Vv 
) , 
i —wds=0,i =1,...,m?;. (6) 
ap; OV 


It is well known that harmonic functions have infinite smoothness. Moreover, as the 
elements of W have a vanishing Neumann derivative on 02, the “variation” of w 
on dQ can only be caused by their behavior near the boundary of D — unless the 
boundary of Q is non-smooth. In other words, the (local) variation of the trace of 
some function w € W is an indicator for the (local) width of the domain Q\D. In 
fact, as we will show next, it is possible to characterize D completely, if the set of 
all traces of W on 0Q were known. (For one insulating inclusion, it is even known 
that the trace of one single potential w € W is enough to identify D, cf., e.g., [16]. 
For conducting obstacles, with known conductivity, the corresponding uniqueness 
problem is still open.) 

To this end, we introduce the Neumann function N(-,z) associated with the 
Laplacian in the domain Q, which is given as the (distributional) solution of the 
problem 


on 02, 


. i) 1 
—AN(x,z) = 6(x-z) inQ, wh =— Bg] (7) 


Lo NG@zjds@) =O, 
where z € Q is kept fixed, and the differential operators act on the x-variable only. 


To achieve a unique solution, we have normalized N(-,z) to have vanishing mean 
on 0Q. The directional derivative 


U(x) = p- grad, N(x, z) (8) 
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with respect to z of N in direction p (of unit length) yields the potential of a dipole 
source in z with moment p in the presence of an insulated boundary 0Q: We refer 
to U, as the dipole potential, tacitly assuming the dipole moment to be fixed. (All 
subsequent results hold true for an arbitrary choice of p € R” with |p| = 1, and it 
appears that there is still space to improve the numerical performance of the method, 
especially in three-space dimensions, provided that this property is exploited in an 
optimal way.) We remark that U, behaves like 


1 (x-z):p = 
Qn |x—-2|? ° ‘ , 
U(x) ~ as X > Z, (9) 
7 1 (x-z):p 
, n=3, 
4n |x —2|3 


and, in fact, U, agrees with the right-hand side of (9) up to a harmonic function. 
This statement holds true for every fixed z € Q. 

Now we are ready to formulate the characterization of the inclusion D as it 
has been established by Briihl in his dissertation [21] (see also [22]) and which 
constitutes the basis for the factorization method. 


Theorem 1. A point z € Q belongs to D, if and only if the trace ¢, = U-z\ae 
coincides with the trace of some potential w € W. 


Proof. First, let z € D. Then the dipole potential U, is harmonic in Q\{z}, ie., in 
Q\D and in a neighborhood of dD. Accordingly U- belongs to Hy (Q\D). As 
N(x, z) has the same Neumann boundary data for any z € R", its directional 
derivative with respect to z has vanishing Neumann data on 0&2. Moreover, 
according to Green’s formula, 


/ eh (10) 
aD; dv 


for every component D; of D which does not contain z; however, as the total flux 
of U; across 0(Q\D) vanishes as well, (10) must also hold true for that component 
D; of D which does contain z. Therefore, U, € VW, and its trace belongs to the 
corresponding trace space. 

Now, let z ¢ D, and assume that the trace ¢, of the dipole potential U; is the 
trace of a potential w € W. As we have seen in the first part of this proof, U, and 
w thus have the same Cauchy data on 0Q, and it follows from the uniqueness of 
solutions of the Cauchy problem for the Poisson equation that U, and w coincide 
in Q\(D U {z}), where both are harmonic. (It is here where the assumption on the 
connectedness of 2\D is needed.) Now, w extends as a harmonic function into the 
point z and, hence, is bounded near z, whereas U, is not, cf. (9). This provides the 
desired contradiction. 

In the last case, where z sits on the boundary of D, we can use the same argument 
as before to show that w and U, coincide in Q\D. According to (6), U, must 
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therefore have a finite H!-norm on Q\ D, which contradicts the asymptotic behavior 
(9) near z € dD. (This argument requires the Lipschitz continuity of dD, because 
this assumption makes sure that we can find an open-cone C C Q\D with vertex in 
z and, hence, that the integral /, |grad U,|?dx is unbounded.) | 


It turns out that the potentials w = uo — uy, which provide the given relative data, 
have additional features that are not captured by the description of the set W of (6). 
For example, if the boundaries of the domains D; are smooth, then the potential 
ug of (2) can be extended by reflection to a certain subset of D, showing that w 
has a harmonic extension to a larger domain than just Q\D (see [54]; Appendix). 
Therefore, the space spanned by the relative data is smaller than the trace space 
of W in general. Still, there is a means to deduce this trace space from the given 
relative data — and the appropriate tool is the factorization method. 

At this point we deviate from the usual presentation of the factorization method 
to opt for a more elementary derivation of the main results. Instead of the usual 
factorization of the data map in three factors as in (1), we follow the approach in 
[23] and factor the relative data in only two parts, namely, 


Ao — Aq = K*K, (11) 
where K* is an appropriate adjoint of the operator K given by 


Ug — uy in Q\D, 


en 12 
Ci; — uy in D;,i =1,...,m, ( ) 


K: fr 
and the real numbers c; in (12) are the means of the potential uo at the boundaries 


of the insulating inclusions, i.e., 


1 


ee uo ds, i=l,...,m. (13) 
|D;| OD; 


Cj 


We claim (see Theorem 2 below for a proof) that K is a continuous operator from 
LZ (dQ) to V, where 


X= WV: QR: vio € HO(Q\D),v| p € H'(D), 


i lds = 07 = 1... (14) 
Dj 


In this definition, again, the subscript © indicates that any v € ¥ is required to have 
vanishing mean on 0Q, and 


[’] = vt lap — lap 


denotes the jump of v across the boundary of the inclusion(s), defined in the 
appropriate trace spaces. Here and below we denote by vt and v~ the restriction 
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of a generic element v € ¥ to Q\D and D, respectively. We equip V with the inner 
product 


wx = / grad v- grad wdx 
Q\aD 


= / gradv -gradw dx + / grad vt - grad wT dx, (15) 
D Q\D 


which turns ¥ into a Hilbert space. Take note that Hy (QQ), i.e., the set of all 
functions from H'(Q) with vanishing mean on dQ, is a subset of 2. 


Lemma 1. LetK C X be the set of all elements w € X that are harmonic in Q\dD 
and satisfy 


0 
—w=0on0Q and cay =0on 0D. 
dv dv 

Then K is the orthogonal complement of HY(Q) in X. 


Proof. Using Green’s formula for any v € H. 5(Q2) and any w € * that is harmonic 
in Q\dD, we obtain 


0 dwt ow” 
/ grad v-grad wdx =f vor as — f yds + | yo" ds 
Q\aD aa. Ov ap. Ov ap OV 
ow dw 
= — ds — — | ds, 
[o> fol 


as v has a well-defined unique trace on 0D. Now, if we choose w € K, then both 
integrals vanish, and hence wv with respect to the scalar product in Vv. 
Vice versa, pick w € ¥ from the orthogonal complement of H, i (Q2), and let v be 


(16) 


a C® function with compact support in 2\D, then Green’s formula yields 


0 
/ wAvdx = / w— as— | i as | grad w- grad vdx 
Q\D aq Ov ap Ov Q\D 


0 0 
= / wo as— | wo as— [ grad w- grad vdx, 
aq Ov ap Ov Q\ aD 


and all three integrals in the bottom line are zero by construction. Thus, w is 
harmonic in Q2\D according to Weyl’s Lemma. The same kind of argument also 
shows that w is harmonic in D. Accordingly, as above, (16) holds true for any v € 
H. 5(2). where now the left-hand side of (16) is zero because of the orthogonality. 
A standard variational argument then shows that the normal derivative of w on dQ 
and the flux of w across dD must vanish. a 
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We briefly mention that every potential w from W of (6) has a unique continua- 
tion to a potential w € K, and the restriction of a nontrivial element from K to Q\D 
is a nonzero element from W. Accordingly, the set of traces on dQ of potentials 
from W and K, respectively, are the same. 


Theorem 2. The operator K : L4 (0) — & defined in (12) is bounded, injective, 
and its range lies dense in the subset K introduced in Lemma 1. The adjoint operator 
K*:X%*> L3 (8) satisfies 


K*y — viag, ve K, 
= FG. ve HA(Q). 


In particular, there holds K* K = Ag — Aj, i.e., (11). 


Proof. We recall that the two Neumann problems (2) and (5) have well-defined 
unique solutions uo and uy in the space H. 6(Q\D) and 7. Ps (Q2), respectively, which 
are given by the corresponding weak formulations 


i; grad uo: grad vodx = i vo ds for every vo € Hy (Q\D), 
Q\D aa (17) 


i grad uy - grad vdx = i; fvds for every v € H3(Q). 
Q aQ 


Moreover, the two solutions depend continuously (in H'!) on the given boundary 
data f € LZ(8Q). Accordingly, w = Kf is a well-defined element of ¥ and K a 
bounded linear operator from LZ (8Q) to V. The jump condition fj, [w]ds = 0 is 
a consequence of the definition (13) of c; and the uniqueness of the trace of uy on 
oD. 

Now, choose any f € LZ (0Q), and denote by uo and uy the corresponding 
solutions of (2) and (5). As in the definition of Kf, we can extend up to a function 


in ug in Q\D, 
uy = Pee 
c; inD;,i=1,...,m, 


in ¥, such that Kf = to — uy. First, for v € HY(Q), we have 


(Kf. v)x = (to, v)e — (un, v)x 


= 7 grad ug: grad vdx— i grad uy - grad vdx = O 
Q\D Q 
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by virtue of (17), and, hence, R(K) 1H. 9 (2). It thus follows from Lemma | that 


R(K) Cc K and N(K*) = Rik) >) HY(Q) and, in particular, that K*v = 0 for 
every v € HY (Q). 
Second, for v € K, we compute 


(Kf. v)x = (to, v)x — (U1. v)x = (to, v)x, 


since uy and v are orthogonal to each other according to Lemma |. Together 
with (17) thus follows that 


(Kf, v)x = / grad ug- grad vdx = i; fvds = (f,V)12(aa): 
Q\D aQ 


ie., that K*v = vlag. In particular, for v = Kf = tig — uz € K, we obtain 
K* Kf = K* (ito — ux) = (uo — U1) ap, 


and, hence, the assertion (11) follows, cf. (4). 

Assume now that 7(K) were not dense in K. Then there is some 0 #4 ve KN 
R(K)- = KON(K*), and since 0 = K*v = vlag, this function v has vanishing 
Dirichlet boundary values on 0&2. Moreover, as v belongs to K, it is harmonic in 
Q\ D with vanishing Neumann boundary values on 0Q (see Lemma 1). Thus, vt = 
Vlo\D = 0 because of the unique solvability of the Cauchy problem for harmonic 


functions. Using Lemma | once more, it follows that v- = v|p is also harmonic 
with vanishing Neumann boundary values on 0D, and, hence, v- is constant on 
each D;, say Vv |p, = v;, i = 1,...,m. Since fyp,[v]ds = —v; |dD| and as v 


belongs to V, these constants must all be zero. This is a contradiction to v ¥ 0, and, 
hence, ?(K) is dense in K. 
Finally, to show injectivity of K, we assume Kf = 0 for some f € L202). 


Then up = uy in Q\D, and uy = c; in D;, i = 1,...,m. Since uy is harmonic 
in all of the domain Q, the field must be constant in Q (principle of unique 
continuation), and the flux f = du /dv vanishes on 0Q. a 


This theorem — together with Lemma | — reveals that the range of K* consists 
of all traces of potentials w € K, whereas the range of Ag — Ay only consists of a 
dense subset of this set. Accordingly, we need to find a way to deduce the range of 
K* from the given data to decrypt the information hidden in these traces according 
to Theorem 1. 

To this end we exploit the so-called Picard criterion, a formulation of which can 
be found in the appendix (Theorem 25) for the ease of completeness. The Picard 
criterion is based on the singular value decomposition of the operator K, which is 
largely equivalent to the spectral decomposition of the operator K* K = Ag — Aq. 


Corollary 1. The operator Ag — Ax is a compact and self-adjoint operator from 
LZ (AQ) into itself. As such, LZ (dQ) has an orthonormal eigenbasis { f;} and 
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associated eigenvalues A ;, such that 
(Ao— Aa) fj =Aj fj, n €N. (18) 


These eigenvalues are positive and converge to zero as n — oo. Throughout we 
shall assume that they are sorted in nonincreasing order. 


Proof. That Ag and Ay are compact operators can be seen from the fact that the 
trace space of H!(Q\D) on dQ, i.e., H'/?(0Q), is compactly embedded in L?(Q). 
Accordingly, the difference operator Ag — Ay is compact as well as self-adjoint, as 
follows readily from (11). One can thus find an orthonormal eigenbasis of Ag — 
Ay, and the associated eigenvalues converge to zero for 7 — oo. It remains to 
prove that they are all positive; this follows from (11) and the injectivity of K by 
Theorem 2. a 


As we have mentioned before, a point z € Q belongs to D, if and only if the 
trace @, of U; is the trace of a potential in K, i-e., if it belongs to the range of K™*. 
As we show in the appendix, cf. Corollary 3, this can be tested in the following way. 


Theorem 3. Let { f;} and {A ;} be the eigenbasis and eigenvalues of Ag— Ax. Then, 
for any point z € Q., 


fore) ; 2 
zeDe> Ie Hees! Ee) < 00 (19) 
J 


n=1 


with 6, = U-;|ag from (8). 


Remark I, With the notations 1/oo = 0 and signa = 


ca ae : for any 


» A=NV, 
a € C, we note that 


-1 
2 
Xp(z) = sign >, eae , zEQ, 
j 


a 


is the characteristic function of D. In particular, this result provides a constructive 
proof of the uniqueness of the inverse problem. 


Conducting Obstacles 


Next, we turn to the case of anisotropic conducting obstacles. To this end we assume 
that for each x € Q the conductivity o(x) is a real, symmetric positive definite 
n X n-matrix, measurable and essentially bounded as a function of x and that the 
associated quadratic form is bounded from below by some positive constant c > 0, 
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1.e., 


p:(a(x)p) = ¢ for almost every x € D and every p € R" with |p| = 1 and 
o(x) = 1 onQ\D, 

(20) 
where D denotes the obstacles, which are assumed to have the same topological 
properties as before. Another assumption that seems to be necessary for the validity 
of the factorization method is that 


Pp: (o(x)p) <« <1 forevery p € R” with |p| = 1, and almost every x € D, 
(21) 
which states that the background conductivity of the object is strictly larger 
than within the inclusions. Instead of (21), one can alternatively require that the 
conductivity within the inclusions is strictly larger than in the background, with 
straightforward modifications of the analysis; however, we will stick to the above 
assumption for the ease of simplicity. We mention that the assumption that the 
background conductivity be strictly larger (or smaller) than within the object can be 
relaxed to just being larger (or smaller), for the prize that the outcome of the method, 
is unspecified for sampling points right on the boundary of the inclusions, cf. [42]. 
However, it is an open problem whether the factorization method is applicable, 
if inequality (21) holds in some obstacles, while p-(o(x)p) = y > 1 in other 
inclusions; numerically, the method does not seem to deterior in this “mixed case.” 
With conducting obstacles, the potential corresponding to a boundary current 
feé LZ (@Q) is given as the (weak) solution u € Ay (Q) of the boundary value 
problem 


0 
div(ogradu) =O in, —u=f ondQ, / uds = 0, (22) 
dv aQ 


which replaces the model (2) from section “Impedance Tomography in the Presence 
of Insulating Inclusions” above. Accordingly, we denote by A the Neumann- 
Dirichlet map associated with (13),i.e,A : fH g =Ulag. 

As before, the corresponding inverse problem is to determine the shape of the 
obstacles D from the relative data A — Ay. Here, again, Ay corresponds to the 
“unperturbed” case 0 = 01 = | everywhere in Q. 

We mention that the problem whether not only D but the conductivity o itself 
is uniquely determined by these data is completely settled when n = 2 — as long 
as o is isotropic — cf. [14]. For n = 3, this question is still open for general scalar 
L® — conductivities. Partial answers are known; we refer to chapter > Electrical 
Impedance Tomography. However, the set D is uniquely determined as we will see 
below in Theorem 7. 

Now we proceed to derive a factorization of A — A, in three factors as in (1), 
Le., 


A — Aq = AGA*. (23) 
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To this end we imagine the effect of a virtual source g on the boundary of the 
obstacle D, given that the boundary of the object Q is insulated. The corresponding 
potential v is the solution of the boundary value problem 


Av=0 inQ\D, ae g ond, 
—v=0 ondQ, } vds = 0. 
dv an 


Recall that the normal vector v on dD has been fixed to point into the interior of 
Q\D, and therefore the minus sign in front of the normal derivative on dD reflects 
the fact that is considered to be a source and not a sink. We will require that this 
source has vanishing mean on each connected component D; of D, i.e., 


yg €H,'?(aD) =\o eH '(aD): | ods =0,i =1,...,m$, (25) 
aD; 


where the integrals have to be interpreted as dual pairings between H~!/? functions 


and the unit constant from H'/?. For later use we remark that the dual space of 


Ay M *(D) can be identified with the subspace 


Hi” (aD) = Wena): [ wds =0,i =1,...,m (26) 
aD; 


of H'/*(aD). 
Associated with (24), we define the operator 


H,'?(aD) > L3,(82), 


Q re vise, 


A: (27) 


and remark that the adjoint operator A* : Ls (0Q) > H a! *(aD) of A is easily seen 


tomap f € EZ (89) onto the trace of the solution uo of (2) on the boundary of the 
obstacle — after an appropriate renormalization of this trace on each component dD; 
of dD. More precisely the following holds 


(A* f)(x) = up(x)-—c; forx € dD;, i = 1,...,m, (28) 


with c; as in (13). 
In order to establish (23), it remains to determine the operator G in the middle. 
We define G via the weak solution w of the diffraction problem 
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0 
div(o gradw) = 0 in Q\dD, —w=0 ondQ, wds = 0, 
dv aQ 


[wlan = ¥. [v-(o grad w)Jap = 0, 

(29) 
and the solution wz of the corresponding problem with o replaced by one every- 
where. Again, the normal v on dD is pointing into the exterior of D. Note that 
when o = | throughout all of Q2, then the corresponding solution wy of (29) can be 
represented as a modified double-layer potential with density w and the Neumann 
function for the Laplacian as kernel, i.e., 


“wes i SNe. (Ids), x € QNOD. 


For a general conductivity tensor, the weak form of (29) is obtained by integrating 
the differential equation against any test function v € H'(Q) and using partial 
integration, which yields 


i grad w: (o grad v) dx = 0 for every v € H'(Q). (30) 
Q\aD 


Now we can make the ansatz w = wy + W with w € H'(Q) to rewrite this as a 
standard variational problem in H!(Q). Find w € H!(Q) such that 


/ grad w-(ograd v) dx = -| grad wy - (ograd v)dx 
Q Q\aD 


for every v € H'!(Q). From this, it follows readily that problem (29) has a unique 
weak solution in H'!(Q\dD), provided that y € H'/*(4D), i-e., that y belongs to 
the trace space of H'(D). In accordance with the definition of A*, however, we will 
restrict y to He (aD). 

The flux of w and wy, across dD is well defined in H~'/?(0D), cf., e.g., [45, 
Thm. 2.5], and there holds 


—wy, ds 


i (wt —whyas= | v-(o grad w_)ds — 
ap; 9 aD; 


ap; OV 


a / div(o grad w)dx -{ Aw,dx = 0. 
Dj Dj 


We can therefore define the bounded operator G in the following way: 


Hi? (aD) > H,'(aD), 
(31) 
wv ae —(wt Wy tla. 


ov 
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Theorem 4. With A and G defined as above, the difference A — Ay of the two 
Neumann-Dirichlet operators associated with (22) and (5), respectively, satisfies 


A — Aq = AGA*. 


Proof. Consider an arbitrary element f € LZ (0Q) and the corresponding function 
w = A* f, which satisfies 


Wlap; = uolap; — Ci. 


where up is given by (2), and c; is as in (13). The function w belongs to H.(aD), 
and it is easy to verify that the associated solution wz of (29) — where o is replaced 


by one — is given by 


Ug — uy in Q\D, 
Wi1= s . 
CG —uy inD;,i=1,...,m, 


where uy is the solution of (5). Similarly, the solution w of (29) is given by 


u—u inQ\D, 
w= ; . 
cq —u inD;,i=1,...,m, 


with u from (22). Accordingly, wt — wf = uf —u*, and hence, 


a 
= GA" f = — (uf - ut) 
ov aD 


If we insert this particular source term ¢ into (24), then we conclude readily that the 
associated solution v of (24) is given by v = ut — Hy It thus follows from (27) that 


AGA* f = Ap = g- 81 = (A- Ax) f 
as required. o 


At this occasion, we recall that every function w € W of (6) has a well-defined 
normal derivative g € Hy yf “( dD) at the inner boundary 0D and, hence, solves the 
corresponding boundary value problem (24). And vice versa, the solution of (24) 
for any gy € Hy ue *(aD) belongs to W. Thus, we can reformulate Theorem | as 


follows. 


Theorem 5. A point z € Q belongs to D, if and only if the trace $, of the dipole 
potential U, in z, defined by (8), belongs to R(A). 


As in the insulating case, it remains to derive a constructive algorithm to test 
whether the trace of some dipole potential belongs to R(A) or not. The next step on 
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our way towards this goal is an investigation of the functional analytic properties of 
the operator G. In the following, we will often consider operators acting between 
a reflexive Banach space X and its dual space X*. We will denote the action of an 
element £ € X* onanelement yw € X by (€, y) and the pair of spaces by (X*, X) 
in order to indicate that the first argument belongs to X* and the second to X¥. A 
particular example is the Sobolev space Hy! *(aD) with dual space H, os *(aD). 
Theorem 6. The operator G : Hy! (aD) > H,'°(aD) is self-adjoint (i.e., 
G coincides with G* : H,!?(8D) > Hy’? (aD) if the bi-dual of H4!?(aD) is 
identified with itself) and coercive, i.e., there exists y > 0 with 


(GHW) =v UlWline@p) for ally € He? (AD). (32) 


Here, {-, -) denotes the dual pairing in the dual system (H='(aD), Hy! (aD). 


Proof. The proof proceeds in a couple of steps. 

1. At first we establish the symmetry of G. Take any y and W from Hy! *(aD), 
define w and wy as in the proof of Theorem 4, and — using w instead of w in (29) 
— define w and wy accordingly. Then we conclude that 


~ 0 
+ + 
—w'ds — —w, d. 
- ay S i Ss 


aD aD ) 
ot O 4: a _ 
= w' —w' ds — w (v-(o grad w_ ))ds 
aD dv aD 


Now we can use (29) and apply Green’s formula in D or Q\D, respectively, in 
each of these integrals (care has to be taken concerning the orientation of the 
normal on dD), to obtain 


(64.0) =- f 


grad w- grad wdx — / grad w- (ograd w) dx 
Q\D D 


+ / grad wy - grad wadx + i grad wy - grad wydx 
Q\D D 
a / grad wy - grad wydx — / grad w- (ograd w) dx, 
Q\aD 


Q\aD 
(33) 
from which the symmetry of G is obvious. 
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2. Turning to the coercivity assertion (32), we fix some y € H, i / °(aD) and employ 
the weak form (30) of (29) with v = w—wy € H'(Q). Starting from (33) with 
w = w, we thus obtain 


(Gy.¥) = [ era wala — / 


grad w- (ograd w) dx 
Q\aD 


= / |grad wy |?dx — / grad w- (ograd w) dx 
Q\aD Q\aD 
+ 2 | grad w- (ograd (w — wz)) dx 
Q\aD 
= / |grad wy |*dx + i grad w- (ograd w) dx 
Q\aD Q\aD 
- 2 | grad w- (ograd wy) dx 
Q\aD 
= / grad wy -((1 — o) grad w,) dx 
Q\aD 
+ / grad (w — wy): (ograd (w — wy)) dx 
Q\aD 
> / grad w, -((1 — o)grad wy) dx. 
Q\aD 


The integrand of the last integral vanishes in Q\D and can be bounded in D 
from below using the restriction (21) on the conductivity. Accordingly we have 


(GW.v)>( -«) f |grad w_|7dx. (34) 
D 
3. To accomplish the proof of (32), we need to show that 
I|grad wy ||72¢p) 2 ¢ Wil ap) (35) 


for some constant c > 0. Assume the contrary: let yw) € Hy! (aD) and 
the corresponding wi ’ be such that lw || H'2(ap) = 1 for every j and 
that | grad wi , 


H'(Q\@aD) as 


>», Converges to zero as j tends to infinity. Define wi ie 
L2(D) 
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with 


1 Ay 
Gi) _ ( :) - 
C; w ds, Ueno. 9 Mh: 
[Di] Jap, \* 


~ (7) 


Then we |p; has vanishing mean on 0D,, and | grad wy — 0 for every 


L?(Dj) 
i=1,...,mas j — oo. By virtue of the Poincaré inequality, this implies that 


we ) —_ to zero in H'(D). From (29), thus follows that the normal derivative 


Jw ) at aD (from either side) tends to zero in H~'/?(0D) and, hence, that 
wi / lo\p converges in H '(Q\D) to the solution of the homogeneous Neumann 


problem, normalized at the outer boundary. In other words, we ) converges to 


zero in H'(D) and in H'(Q\D) as j — oo. Recurring to (29) once again, we 
observe that 


j GV) GZ) Gi) ~ (j) 
WO law, + oF" = [i Ino, i a [ ie 20) 


1/2 


and since w\) € H,’° (dD), it follows by integration over dD, that 


1 : 
P= Wq 
“i = op, l we lo, 4 “apie 


wi?| 
ds > 0 
ey “1 loo, > 


as j runs to infinity. Inserting this into (36), we conclude that 


WO lop, = ae ~c) +0, j>0 


in H'/*(8D;), i = 1,...,m, but this contradicts eaceleo = 1. 
Therefore, (35) is true for some c > 0 and every yw € HL’ (aD), and hence, (32) 
follows from (34) and (35). a 


By virtue of Theorem 6, all assumptions of Corollary 5 are satisfied for the 
factorization of the relative data A — Ay established in Theorem 5. Therefore, we 
can now conclude the main result of this section. 


Theorem 7. Let z € Q and $, be defined as before. 
2 
3 |(¢-, £7) 12002)| aa 
ij 


Then: zEeDe> 


where f; and i; are the orthonormal eigenfunctions and eigenvalues of A — Ay. 
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Local Data 


It is an important feature of the factorization method that it can be easily adapted to 
applications where the given data correspond to what is called the local Neumann- 
Dirichlet map A‘. This is the map that takes Neumann boundary values supported on 
some relatively open subset [ C dQ only and returns the corresponding boundary 
potentials on the very same subset (normalized to have vanishing mean, say). The 
local Neumann-Dirichlet map occurs whenever part of the boundary is inaccessible 
to measurements, in which case I’ corresponds to that part of the boundary of Q 
where electrodes can be attached. Mathematically, the local Neumann-Dirichlet map 
can be interpreted as a Galerkin projection 


A‘ = PAP* (37) 


of the full Neumann-Dirichlet map, where 


2 2 
a L3(@Q) > Pe), cc 


g > glr— py Jr eds, 


and P* is its L? adjoint, i.e., 


x, _§ff onl, 
Prf=yi on dD\T. 


From Theorem 4 we immediately conclude that if the conductivity distribution 
satisfies (20) and (21), then the difference of the two local Neumann-Dirichlet maps 
A‘ and A‘ can be factorized in the form 


A‘ — A‘ = (PA)G(PA)* 


with A and G as before. Moreover, the coercivity of G allows a constructive way to 
check whether a given function belongs to 7?(PA), considered as an operator from 
H,'/ *(aD) to LZ). Note that it is obvious from Theorem 5 that the function P¢, 
belongs to R(PA) when z € D; the converse statement requires a little more effort. 


Theorem 8. Let Tl be a relatively open subset of 0Q, and let P be the projector 
defined in (38). Then z € D, if and only if Pz, € R(PA). 


Proof. According to the definition (27) of A, the test function P¢, belongs to 
R(PA), if and only if ¢, coincides on T’ (up to a constant) with the trace of a solution 
v of (24). In this case, however, the dipole potential U, and the function v are both 
harmonic functions in Q\(D U {z}) and have the same Cauchy data on I (again, 
up to a constant). Now we choose a connected subset Q’ of 2\(D U {z}), whose 
boundary contains a portion of I that is also a relatively open subset of dQ. Then U- 
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and v coincide up to a constant in 2’ according to Holmgren’s theorem and, hence, 
near all of dQ. This shows that ¢, € R(A), and, hence, the assertion follows from 
Theorem 5. a 


Accordingly, if I’ is a relatively open subset of dQ, then Theorem 7 also extends 
readily to the local situation, if the eigenfunctions and eigenvalues of A — Ay are 
replaced by those of A‘ — MG 

Note that Theorem 8 requires that I’ is a relatively open subset of dQ, and 
in fact, the factorization method no longer applies for discrete measurements 
or finitely many boundary currents. Still, this is precisely the situation that is 
encountered in practice, as data are always finite dimensional. Due to the rapid 
decay of the eigenvalues of A — Ay, however, the full relative data can be very 
well approximated by operators of finite rank, such as those corresponding to real 
data (see [54] for detailed numerical examples). 


Other Generalizations 


The Half-Space Problem 

The factorization method can also be applied to a related inverse electrostatic 
problem in full space with near field data, if the same manifold of codimension one 
is used to generate a source and to measure the resulting change of the potential. 
In fact, this problem which has been studied in [52] and [75] is very similar to the 
setting for the Helmholtz equation that we will consider in the following section. 
We also like to refer to [15] where this approach has been applied to some real 
two-dimensional data. 

For quite a few applications, however, the impedance tomography problem is 
more appropriately modeled in a half space, rather than in the full space or within a 
bounded domain. For this setting new difficulties arise, as the data (may) live on the 
entire, unbounded boundary of the surface, which calls for weighted Sobolev spaces 
for an appropriate theoretical analysis. In the sequel we restrict our attention to 
three-space dimensions (n = 3), as the two-dimensional case needs some additional 
attention, cf. [55] and at the same time appears to be less interesting from a practical 
point of view. 

We consider the half-space 2 = {x € R® : v-x < O}, where v € R? isa fixed 
unit vector, which coincides with the outer normal on the hyperplane {x : v-x = 
0}, which is the boundary of Q. The main difficulty in the analysis of this problem 
is that solutions of the corresponding conductivity problem 


a 
div(ogradu) = 0 inQ, apt =f ondQ, (39) 
v 


need no longer belong to L7(Q); instead, one has to resort to weighted Sobolev 
spaces, such as 
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U={ue DQ): (14+ |- |?) 1/7u € L*(Q), |grad ul € L?(Q)}, 


to search for a unique solution of (39). If o is given by (20), then a weak solution 
u € U can be shown to exist provided that f belongs to 


L?"(8Q) ={f (+ |-P)'7 Ff € L2(0Q)}, 


in which case the trace of u belongs to the dual space L”:!(0Q) of L?~!(0Q). Note 
that no normalization of u is required in (39) because solutions in U/ are implicitly 
normalized to vanish at infinity. We refer to [55] for further details about the forward 
problem. 

Within this function space setting, the Neumann-Dirichlet operator is defined 
in a natural way as an operator A : L*~!(0Q) + L*!(@Q), and the difference 
between A and A, (the latter corresponding to the homogeneous half space) admits 
a factorization (23) as before, where now 


4.) He ?@D) > LQ), 
g > Vian, 

and v solves the same boundary value problem as in (24), except for the missing 
normalization over the boundary 02. Furthermore, the self-adjoint operator G is 
defined as before (with the appropriate definition of a weak solution of (29)) and is 
coercive again. 

We emphasize that the dipole potential (8) for the half space is explicitly known, 
i.e., we have (up to a negligible multiplicative constant) 


(x—z)*p 


=r xe dQ. (40) 
Xe ZF 


G(x) = 


With these notations, the characterization of the inclusions can be established in 
much the same way as before (see [55]). 


Theorem 9. A point z € Q belongs to D, if and only if $, of (40) belongs to R(A). 


For real applications, the measuring device will only cover a bounded region 
I Cc dQ. The corresponding local Neumann-Dirichlet operator A‘ can then be 
embedded in the standard L? framework from the previous section, and the usual 
Picard series can be used to implement the range test. For the ease of completeness, 
we briefly mention that for such local data the test dipole ¢, can be replaced by the 
function 


" 1 
d(x)=—,, xrel, 
|x —2| 
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which is the trace of the corresponding Neumann function (again, up to a multi- 
plicative constant), as the latter has a vanishing normal derivative on the boundary 
of the half space. We hasten to add, though, that db: must not be used for full data, 
as it does not belong to L?!(0Q). Numerically, however, this modification of the 
method has no significant benefit. 


The Crack Problem 

Another case of interest are cracks, i.e., lower-dimensional manifolds of codi- 
mension one, that are insulating, say. This setting has important applications in 
nondestructive testing of materials. Consider a domain Q C R”, withn = 2 or 
n = 3 again, and the union & = U"L,&X; C Q of m smooth, bounded manifolds 
(the insulating cracks), such that £; M 2; = O and Q\> are connected. Given a 
boundary current f € LZ (AQ), the induced potential satisfies the model equations 


Aupy = 0 in Q\d, Leer, on &, ae f ondQ, (41) 
dv dv 


and the corresponding Neumann—Dirichlet operator is the map that takes f onto the 
trace of up on 0Q: 


{13(02) > 132), 


A 
f a Uolaa- 


The crack case can be analyzed in a similar way as in section “Impedance 
Tomography in the Presence of Insulating Inclusions”, cf. [23], using a factorization 
A — Ay = K*K, where K is almost identical to the operator in (12), except that 
it maps into H!(Q\>). There is a more important difference, though. As the crack 
has no interior points, the range test will always fail with the hitherto used test 
function ¢,, as the dipole singularity of U, is too strong to belong to H'(Q\%), 
even when z € &. To detect a crack, we therefore need to construct a new test 
function by integrating the function ¢, over z along some “test arc” (in R*) or some 
“test surface” (in R3). 

The range test can then be implemented by placing linear (planar) test cracks in 
different sampling points with various orientations (see [23] for numerical recon- 
structions in two-space dimensions). The amount of work thus grows substantially, 
as we now have 2 degrees of freedom to sample (a test point and a normal direction) 
instead of only one in the previous cases. Also, in a numerical realization, test cracks 
will — at best — only touch the crack tangentially, but in theory this already suffices to 
ruin the range test. It turns out that in practice the usual implementation with the test 
function ¢, performs as good as the more elaborate but expensive variant described 
above. As said before, in theory, ¢, will never belong to the range of K; in practice, 
however, it will “almost” do so, i.e., the Picard series (19) will grow much more 
slowly in the close neighborhood of the crack. 

One-dimensional cracks in three-dimensional objects cannot be reconstructed 
in this way, because the potential does not “see” inhomogeneities of this size. 
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However, one can use an asymptotic analysis similar to the derivation of MUSIC- 
type algorithms that are discussed in section “MUSIC” below. Here we give a brief 
sketch of an argument provided in [47] and refer to this paper for further details. The 
basic idea is that realistic “one-dimensional” cracks in a 3D world are not exactly 
one-dimensional, but better modeled as extremely thin tubular inclusions of small 
diameter 6 > 0. The corresponding relative data As—A4, where Aj is the Neumann- 
Dirichlet operator associated with the tubular inclusion and A, is as usual, turn out 
to satisfy an asymptotic expansion in 6, 


As— Aq = 8M + 0(8”), 


possibly after selecting an appropriate (sub)sequence 6, —> 0. The operator M 
that constitutes the dominating term of this expansion admits a factorization similar 
to (23). In contrast to the MUSIC framework below, this operator has infinite 
dimensional range. Although the operators of the corresponding factorization are 
somewhat different from the ones that we have encountered above, the bottom 
line is the same as for one-dimensional cracks in two-space dimensions. The same 
integrated test function belongs to the range of the operator A of this factorization, 
if and only if the corresponding test arc is part of the crack. The singular value 
decomposition of M can be used to evaluate this test, and in practice this singular 
value decomposition can be approximated by the one of As — Aj, i.e., by the given 
data. 


3 The Factorization Method in Inverse Scattering Theory 


The second part of this chapter is devoted to the factorization method for problems 
in inverse scattering theory for time-harmonic waves. The scattering of an incident 
plane wave by a medium gives rise to a scattered field which is measured “far 
away” from the medium. The factorization method characterizes the shape of the 
scattering medium from this far field information. The measurement operator will 
be the far field operator F which maps the density of the incident Herglotz field to 
the corresponding far field pattern of the scattered field. 

The far field operator F allows a factorization of the form (1) where the operators 
A and G depend on the specific situation. We will discuss two typical cases and 
start with the scattering by a sound-soft obstacle D in section “Inverse Acoustic 
Scattering by a Sound-Soft Obstacle.” This is an example of a nonabsorbing medium 
which is mathematically reflected by the fact that the far field operator is normal — 
though not self-adjoint as for the corresponding problem in impedance tomography. 
It was this example for which the factorization method was developed for the first 
time in [66]. In section “Inverse Electromagnetic Scattering by an Inhomogeneous 
Medium,” we will study the scattering of time-harmonic electromagnetic plane 
waves by an absorbing medium. In this case the corresponding far field operator 
fails to be normal. 

Each case study will start with a short repetition of the corresponding direct 
problem. Then the inverse problem will be stated, and a factorization of the form 
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(1) will be derived. As in impedance tomography, a crucial point is to establish in 
each case a certain coercivity condition for G. In addition, one needs to prove a 
range identity which describes the range of A via the known — possibly non-normal 
— data operator F. 

Here and throughout the following sections, S? = {x € R? : |x| = 1} denotes 
the unit sphere in R?. 


Inverse Acoustic Scattering by a Sound-Soft Obstacle 


This section is devoted to the analysis of the factorization method for the most 
simplest case in scattering theory. We consider the scattering of time-harmonic plane 
waves by an impenetrable obstacle D C IR* which we model by assuming Dirichlet 
boundary conditions on the boundary 0D of D. As before, we assume that D is a 
finite union D = U’_, D; of bounded domains D; such that Di; ND j = @ for 
i # j. Furthermore, we assume that the boundaries 0D; are Lipschitz continuous 
and that the exterior R? \D of D is connected. Finally, let k > 0 be the wave number 
and 


ul (x) = exp(ikx- 6), x € R', (42) 
be the incident plane wave of direction 6 € S*. The obstacle D gives rise to a 
scattered field u’ € C?(IR°\D) M C(R?\D) which superposes u' and results in the 
total field u = u' + u’ which satisfies the Helmholtz equation 
Au+k*u=0 outside D (43) 
and the Dirichlet boundary condition 


u=O0 ondD. (44) 


The scattered field u* satisfies the Sommerfeld radiation condition 


S 


0 _ i = Or) forr = Ix| > 00 (45) 
. 


uniformly with respect to = x/|x| € S*. 

The direct scattering problem is to determine the scattered field u* for a given 
obstacle D C R?, some 6 € S2andk > 0. 

For the treatment of this direct problem, we refer to [36] (see also section 
“Obstacle Scattering” in chapter > Inverse Scattering). There it is also shown that 
the scattered field u° has the asymptotic behavior 


u(x) = PCHAD 08) + Ox[), |x| > 00 (46) 


4r|x| 
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uniformly with respect to = x/|x| € S*. The function uw : S$? — C is analytic 
and is called the far field pattern of u*. It depends on the wave number k, the 
direction 6 € S >, and the domain D. Since we will keep k > 0 fixed, only the 
dependence on 6 is indicated: u = u(x; 6) for x, 6€ S?. 

In the inverse scattering problem, the far field pattern u(x; 6) is known for 
all x, be S? and some fixed k > 0, and the domain D has to be determined. We 
refer again to [36] or chapter > Inverse Scattering for the presentation of the most 
important properties of this inverse scattering problem. The knowledge of u™ (x; 6) 
for all £:6 € §S? determines the integral kernel of the far field operator F from 
L?(S7) into itself, which is defined by 


(Fg)(&) = i u(&; 6)g(0)ds(6) for & € S?. (47) 


The far field operator F is compact, normal (i.e., F commutes with its adjoint F’*), 
and the so-called scattering operator I + oSF is unitary in L?(S7). 

As in section “Conducting Obstacles,” the first step is to derive a factorization of 
F in the form (1). 

The operator A is the data-to-pattern operator which maps f € H'/?(dD) to 
the far field pattern v® of the radiating (i.e., v satisfies the Sommerfeld radiation 
condition (45)) solution v € H].(IR?\D) of 


Av + k*v = 0 in the exterior of D, v = f on oD. (48) 


Here, H' 


loc 


(IR3\ D) is the space of functions v with v| a\p © H '(B\D) for all balls 
B CR’. Existence and uniqueness is assured (see, e.g., [80]; Chap. 9). 


Theorem 10. Define the operator A : H'/*(@D) — L*(S*) by Af = v where 
v© is the far field pattern of the unique radiating solution v € H)\(IR?\D) of (48). 
Then A is one-to-one with dense range, and the following factorization holds 


F = -AS* A*, (49) 
where A* : L?(S?) —~ H~'/?(@D) is the dual of A, and S* : H~'/?(@D) > 


H'/?(aD) is the dual of the single-layer boundary operator S_: H~'/?(aD) > 
H'/?(aD) defined by 


(S—)(x) = / POPC yd5(y), © OD. (50) 


Here, ® denotes the fundamental solution of the Helmholtz equation, i.e., 


exp(ik|x — y|) 


' x yeR xy, (51) 
4n|x — y| 


O(x,y) = 
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and the explicit definition (50) of this operator makes only sense for smooth 
functions . It has to be extended to functionals gp € H~'/?(dD) by a density or 
duality argument. 


Proof. The injectivity of A follows immediately from Rellich’s Lemma (see [36] 
or chapter > Inverse Scattering). The denseness of the range of A can be shown by 
approximating any g € L*(S”) by a finite sum of spherical harmonics to which the 
corresponding field can be written down explicitly. 

To derive the factorization, define the auxiliary operator H : L*(S*) > 
H'/?(aD) by 


(Hg)\(x) = [. g(0) exp(ikx- 6)ds(6) = [eG (x: 6)ds(6), x €aD. 


First we note that u%°(-; 6) = —Au'(-; 6) by the definition of A, and thus by the 
superposition principle, Fg = —AHg for all g € L*(S), i.e., F = —AH. We 
compute the dual H* : H~!/?(dD) — L?(S?) as 


(H*9)(2) = i PO exPI-IkE:y)ds(y), Fe 5? 


The fundamental solution ® has the asymptotic behavior 


exp(ik]|x]) 


P(x, y) = exp(—ikX-y) + O(|x|7), |x| > 00 (52) 


A4r|x| 


uniformly with respect to £ € S* and y € dD and thus has the far field pattern 
© (xX, y) = exp(—ikX-y). Therefore, again by superposition, H*g = ASQ, i.e., 
H = S* A*. Substituting this into F = —AH yields (49). | 


Therefore, F allows a factorization in the form (1) with G = —S*. The most 
important properties of this operator are collected in the following theorem. (For a 
proof see, e.g., [73, 80].) 


Theorem 11. Assume that k? is not a Dirichlet eigenvalue of —A in D. Then the 
following holds: 


* Sis an isomorphism from the Sobolev space H~'/?(dD) onto H'/?(dD). 

* Im(g, Sv) < 0 forall g ¢ H~'/?(dD) with o # 0. Here, (.,.) denotes the 
duality pairing in (H~'/*(0D), H'/?(0D)). 

e Let Si be the single-layer boundary operator (50) corresponding to the wave 
number k = i. The operator Sj; is self-adjoint and coercive as an operator from 
H-'/?(8D) onto H'/?(aD), i.e., there exists co > 0 with 


(pSig) = collglly—2qp) for alle € HD). (53) 
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* The difference S — S; is compact from H~'/?(dD) into H~‘/2(aD). 


From this theorem the following coercivity result can be derived. 
Assume that k? is not a Dirichlet eigenvalue of —A in D. Then there exists c; > 0 
with 


K(eS9)| > cr lelly—@p) forall g € H7/7(aD). (54) 


This establishes the first step of the factorization method. In the second step, the 
domain D is characterized by the range of the operator A. 


Theorem 12. For any z € R°, define the function $, € L?(S*) by 
$,(%) = exp(-ik&-z), % € S?. (55) 
Then z belongs to D, if and only if 6, € R(A). 


Proof. Let first z € D. From (52), we conclude that @, is the far field pattern of 
®(-,z); thus, ¢, = Af, where f = ®(-,z)|an € H'/?(0D). 

Let now z ¢ D and assume, on the contrary, that ¢, = Af for some f € 
H'/?(QD). Let v be as in the definition of Af. Then ¢, = v°°. From Rellich’s Lemma 
and unique continuation, we conclude that ®(-, z) and v coincide in R*\(D U {z}). 
By the same arguments as in the proof of Theorem |, this is a contradiction since v 
is regular and ®(., z) is singular at z. | 


From the factorization (49), we conclude that R(F) C F(A), and thus 
o.€ R(F) => ze D. 


Therefore, the condition on the left-hand side determines only a subset of D. One 
can show, cf. [35], that for the case of D, being a ball, the left-hand side is only 
satisfied for the center of this ball. Nevertheless, the (regularized version) of the test 
. € R(F) leads to the Linear Sampling Method, cf. section “The Linear Sampling 
Method”. 

In the third step of the factorization method, the range 7(A) of A has to be 
expressed by the known data operator F’. This is achieved by a second factorization 
of F based on the spectral decomposition of the normal operator F’. From now on 
we make the assumption that k? is not a Dirichlet eigenvalue of —A in D. Then the 
far field operator is one-to-one as it follows directly from the factorization (49) and 
part (a) of Theorem 11. 

Since F is compact, normal, and one-to-one, there exists a complete set of 
orthonormal eigenfunctions w; € L?(S) with corresponding eigenvalues A jz € 
C, j =1, 2, 3, ... (see, e.g., [88]). Furthermore, since the operator J +ik / (827) F 
is unitary, the eigenvalues A; of F lie on the circle of radius 1/r and center i/r 
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where r = k/(827). The spectral theorem for normal operators yields that F has 
the form 


CO 
Fw = AWW )isavi, WE L7(S?). (56) 
j=! 
Therefore, / has a second factorization in the form 
F = (F*F)'4G,(F*F)'4, (57) 
where the self-adjoint operator (F*F)!/4 : L?(S*) — L?(S?) and the signum 


G2 : L?(S*) > L?(S?) of F are given by 


(F*F)'4y = 0 VAG visas, we L7(S’), (58) 


j=l 


Gow = AG wey, ve L(S?). (59) 


j=l 
Also this operator G2 satisfies a coercivity condition of the form (54). 


Theorem 13. Assume that k? is not a Dirichlet eigenvalue of —A in D. Then there 
exists C2 > O with 


(vy, Gow) 12(52) 


> c2||Wilia(92) forall y € L7(S”). (60) 


Proof. Itis sufficient to prove (60) for w € L?(S7) of the form y = par c;w; with 
IY Wi2¢82) =) |c; |’ = 1. With the abbreviation s; = ;/|A;|, itis 


co co lo e) 
|(Gav, W)12(8| = i as = yale : 
j=1 j=l L?(S2) j=l 


The complex number paied syle; |> belongs to the closure of the convex hull C = 


conv {s; : 7 € N} Cc C of the complex numbers s;. We conclude that 


|(Gov, W)z2(82)| = inf {|z| : z € C} 

for all wy € L?(S7) with |v] 12(2) = 1. From the facts that A; lie on the circle with 
center i/r passing through the origin and that 4; tends to zero as j tends to infinity, 
we conclude that the only accumulation points of the sequence {s;} can be +1 or 
—1. From the factorization (49) and Theorem 11, it can be shown (see the proof of 
Theorem 1.23 of [73]) that indeed 1 is the only accumulation point, i.e., 5; — 1 as 
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j tends to infinity. Therefore, the set C is contained in the part of the upper half disk 
which is above the line / = {15 + (1 —t)1 : ¢ € R} passing through S and 1. Here, 
S is the point in {s; : 7 € N} with the smallest real part. Therefore, the distance of 
the origin to this convex hull C is positive, i.e., there exists c. with (60). 

From Theorem 10 and Eq. (57), the scattering operator F can be written as 


F = AG,A* = (F*F)'4G,(F* F)"4, (61) 


where we have set G; = —S*. Both of the operators G;, j = 1, 2 are coercive 
in the sense of (54) and (60), respectively. By the range identity of Corollary 4, the 
ranges of A and (F* F)!/4 coincide. The combination of this result and Theorem 12 
yields the main result of this section. (To derive the second equivalence of (62), 
Theorem 25 of Picard has been applied.) 


Theorem 14. Assume that k? is not a Dirichlet eigenvalue of —A in D. For any 
z € R’, define again ¢, € L*(S*) by (55), ie., 


(&) := exp(—ikX +z), & € S?. 
Then 


2 
Wi) 12°92 
¥)12(89| e 


26D <=> $,€ R(F*F)'4) > Vo io: aI (62) 
j J 


Here, 4; € C are the eigenvalues of the normal operator F with corresponding 
normalized eigenfunctions yj € i7(8"). 

Formula (62) provides a simple and fast technique to visualize the object D by 
plotting the inverse of the series on the right-hand side. In practice, this will be 
a finite sum instead of a series, but the value of the finite sum is much larger for 
points z outside than for points inside of D. We refer to the original paper [66] for 
some typical plots. 


Remark 2. It is illuminating to compare the presentation in this section with the 
one for impedance tomography from section “Conducting Obstacles”. The relative 
potential u— uz considered there corresponds to the scattered wave u* = u— ul’, i.e., 
the total field minus the incoming field; the incoming field is the potential that is 
induced by the excitation if the background is homogeneous, whereas the total field 
is the corresponding solution in the presence of the scatterer. 

In both cases, the operator that maps the excitation onto the associated “relative 
data” can be factorized in three operators: the one that is applied first, i.e., A*, 
maps the excitation/the incoming field onto the boundary of the obstacle(s), the 
operator A that is applied last, maps appropriate boundary data on the obstacle onto 
the “outgoing field,’ and its measured data. Accordingly, the operator in the middle 
encodes the “refraction” at the obstacle(s). 
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As such, we can view the factorization from impedance tomography as a 
generalization of Huygens’ principle to the diffusion problem (22), although the 
time causality from scattering theory has no apparent physical analog in stationary 
diffusion processes. 


Inverse Electromagnetic Scattering by an Inhomogeneous Medium 


This section is devoted to the analysis of the factorization method for the inverse 
scattering of electromagnetic time-harmonic plane waves by an inhomogeneous 
nonmagnetic and conducting medium. Let k = w,/eofo > 0 be the wave number 
with angular frequency a, electric permittivity ¢9, and magnetic permeability jo in 
vacuum. The incident plane wave has the form 


, A ’ 1 ‘ 
H' (x) = p exp (ikO-x), E'(x) = “Te H' (x), (63) 


for some polarization vector p € C* and some direction 6 € S? such that Pp: 6=0. 
This pair satisfies the time-harmonic Maxwell system in vacuum, i.e., 


curl E! —iopoH' =0 inR?, (64) 
curl H' + iwey9E' =0 inR?. (65) 


This incident wave is scattered by a medium with space-dependent electric per- 
mittivity ¢ = e(x) and conductivity 0 = o(x). We assume that the magnetic 
permeability jz is constant and equal to the permeability {19 of vacuum. Furthermore, 
we assume that € = &9 and o = O outside of some bounded domain. The total 
fields are superpositions of the incident and scattered fields, i.c., E = E' + E° and 
H = H' + H’ and satisfy the Maxwell system 


curl EF —iwuoH =0 in R’, (66) 


cullH +iweE =oE inR’. (67) 


Also, the tangential components of £ and H are continuous on interfaces where o 
or € are discontinuous. Finally, the scattered fields have to be radiating, i.e., satisfy 
the Silver-Miiller radiation condition 


1 
JboH’ (x) x X —.feoE* (x) = O (<=) as |x| > 00 (68) 


uniformly w.r.t. = x/|x| € S*. The complex-valued relative electric permittivity 
€, is defined by 
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ex) | 2) 


(x) = (69) 
E0 WEQ 
Note that ¢, = 1 outside of some bounded domain. Equation (67) can then be 
written in the form 
curl H +iwege,E =0 in R?. (70) 


It is preferable to work with the magnetic field H only. This is motivated by the 
fact that the magnetic field is divergence free as seen from (66) and the fact that 
div curl = 0. In general, this is not the case for the electric field £. Eliminating the 
electric field E' from the system (66) and (70) leads to 


1 
curl Ee a] -kRH=0 inR’. (71) 


r 


The incident field H’ satisfies 
cur’? H'—k*?H' =0 inR’. (72) 


Subtracting both equations yields 


1 ; 
curl leu i] —k? H® =curl [qcurl A’) inR’, (73) 


r 


where the contrast g is defined by g = 1 — 1/e,. The Silver-Miiller radiation 
condition turns into 


1 
curl H*(x) x X —ikH’(x) = O (<2) , |x| > oo. (74) 
x 


The continuity of the tangential components of / and H translates into analogous 
requirements for H* and curl H®. 

It will be necessary to allow more general source terms on the right-hand side 
of (73). In particular, we will consider the problem to determine a radiating solution 
v € Ajge(curl, R’) of 


1 
curl | ou 4 —k?v = curl f in R? (75) 


Er 


for given f € L?(R*)> with compact support. (For any open set D the space 
L?(D) denotes the space of vector functions v : D — C? such that all components 
are in L*(D).) The solutions v of (75) as well as of (71) and (73) have to be 
understood in the variational sense, i.e., v € Ajoc(curl, R*) satisfies 
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/ | zcuslv-curl —k?y. v| dx = i f -curlydx (76) 
R3 R3 


or 


for all Y ¢€ H(curl, R*) with compact support. For any domain Q, the Sobolev 
space H(curl, Q) is the space of all vector fields v € L*(Q)> such that also curl v € 
L?(Q)>. Furthermore, Hj,-(curl,R*) = {v: v|g_ € A(curl, B) for all balls BC 
R3}. 

Outside of the supports of e, — 1 and f, the solution satisfies curl’v — k?v = 0. 
Taking the divergence of this equation and using the identities div curl = 0 and 
curl? = —A + grad div, this system is equivalent to the pair of equations 


Av+k?v =0 and divv = 0. 


Classical interior regularity results (cf. [80] combined with [36]) yield that v 
is analytic outside of the supports of ¢,— 1 and /. In particular, the radiation 
condition (74) is well defined. 

There are several ways to show the Fredholm property of Eq. (75). We refer to 
[81] for the treatment by a variational equation with nonlocal boundary conditions 
or to [73] for a treatment by an integrodifferential equation of Lippmann-Schwinger 
type. 

The question of uniqueness of radiating solutions to (75) is closely related to 
the validity of the unique continuation principle. It is known to hold for piecewise 
H6lder-continuously differentiable functions ¢, (see [81]). 

As in the case of the Helmholtz equation, every radiating vector field v 
satisfying curl’v — k?v = 0 outside of some ball has the asymptotic behavior 


exPGE IAD cog) 4 OL|x|-2), [x] > 00, 


v(x) = 


An |x| 


uniformly with respect to = x/|x| € S (see again [36]). The vector field v 
is uniquely determined and again called the far field pattern of v. It is a tangential 
vector field, ie., v°° € L?(S?) where 


L?(S?) = {w € L?(S7)? : w(&)-% = 0,% € S73. 


The inverse problem is to determine the shape D of the support of the contrast 
q from the far field pattern H®(&: 6, p) for all £,6 € S? and p € C? with 
p-6 = 0. Because of the linear dependence of H® on p, it is sufficient to 
know H°® only for a basis of two vectors for p. As in impedance tomography, 
the task of determining only D is rather modest since it is well known that one 
can even reconstruct g uniquely from this set of data (see [38]). However, the proof 
of uniqueness is nonconstructive, while the factorization method will provide an 
explicit characterization of the characteristic function of D which can, e.g., be 
used for numerical purposes. Also, the factorization method can — with only minor 
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modifications — be carried over for anisotropic media (as in section “Conducting 
Obstacles”) where it is well known that ¢, can only be determined up to a smooth 
change of coordinates. 

For the remaining part of this section, we make the following assumption: 
Assumption 12. Let D C R° be a finite union D = U'"_, D; of bounded domains 
Dj such that DM D; = QO fori # j. Furthermore, we assume that the boundaries 
dD; are Lipschitz continuous and the exterior R>\D of D is connected. Let &, € 
L™(D) be such that the values ¢,(x) vary in a compact subset of the half disk 
{2 €C: (Rez—1/2)? + (Im z)? < 1/4, Imz > 0}, and that for every f € L?(R*)? 
with compact support there exists a unique radiating solution of (75). 

We extend £, by one outside of D and define the contrast by q = 1 — 1/é,; thus, 
Im q => Oand Req < —y|q| on D for some y > 0. 


Condition (3) is, e.g., satisfied for Hélder-continuously differentiable parameters 
é and o (see [81]). 
The far field operator F : L?(S*) — L?(S?) is defined as 


(Fp)(&) := i. H®™(&:0, p(@))ds(@), & € S?. (77) 


F is a linear operator since H™ depends linearly on the polarization p. 

The first step in the factorization method is to derive a factorization of F in 
the form F = AT* A* where the operators A : L?(D)? > L?(S?) and T 
L?(D)> - L?(D)* are defined as follows. 

The data-to-pattern operator A : L?(D)? — L?(S”) is defined by Af := v®, 
where v® denotes the far field pattern corresponding to the radiating (variational) 
solution v € Hj,.(curl, R?) of 


1 

curl | eu | —k?v = curl ay in R°. (78) 
oF vial 

Again, the contrast is given by gq = 1 — 1/e,. We note that the solution exists by 

part (3) of Assumption 12. 


The operator T : L*(D)* + L?(D) is defined by Tf = (sign 7) f — 
J|q| curl w|p, where w € Aj-(curl, R*) is the radiating solution of 


curl’ w — k?w = curl [Lv als | in R?. (79) 
The solution exists and is unique (see, e.g., [73]). 
Theorem 15. Let Assumption 12 hold. Then F from (77) can be factorized as 


F = AT*A’*, (80) 
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where A* : L?(S”) > L?(D)? and T* ; L?(D) + L?(D)? denote the adjoints of 
A and T, respectively. Furthermore, A* is injective. 


For a proof of this and the following result, we refer to [73]. 


Remark 3. The solution w of (79) can be expressed in the form (see [73]) 


w(x) = cul I VEO FO). y)dy, x ER’, 


which yields an explicit expression of 7. 
The following theorem corresponds to Theorem |1 and collects properties of the 
operator T needed for the analysis of the factorization method. 


Theorem 16. Let the conditions of Assumption 12 hold, and let T : L?(D)> > 
L?(D)°> be defined above. Then the following holds: 


(a) The imaginary part ImT = x(T — T*) is non-positive, i.e., 
Im(7f, f)22(p)3 < 0 forall f € L?(D)’. 


(b) Define the operator To in the same way as T but for k = i. Then —Re Tp is 
coercive, and T — Ty is compact in L?(D)°. 
(c) T is an isomorphism from L*(D)> onto itself. 


As in section “Inverse Acoustic Scattering by a Sound-Soft Obstacle” we first 
characterize the domain D by the range R(A) of A. The proof of the following 
result can again be found in [73]. 


Theorem 17. Let the conditions of Assumption 12 hold. For any z € RR? and fixed 
p € C’, we define ¢, € L?(S*) as the far field pattern of the electric dipole at z 
with moment P, i.e., 


$:(X) = —ik(% x p)exp(—ik&-z), % € S?. (81) 


Then z belongs to D, if and only if ¢, € R(A). 

In contrast to the data operators Ag — Ay or A — Ay of Sect.2 or the far field 
operator F' of section “Inverse Acoustic Scattering by a Sound-Soft Obstacle”, the 
far field operator for absorbing media — as in the present case — fails to be normal 
or even self-adjoint. Therefore, the approaches of the previous sections — i.e., the 
application of the range identities of Corollaries 4 and 5 — are not applicable. 
However, application of Theorem 27 to the far field operator F from L?(S7) into 
itself and the operator G = T* : L?(D)* + L?(D)? yields the characterization 
of D via an auxiliary operator 


Sampling Methods 627 
Fy =|Re F|+ImF, (82) 
cf. (109), which is easily obtained from the given far field data. 


Theorem 18. Let the conditions of Assumption 12 hold. For any z € IR, define 
again $, € L?(S*) by (81). Then, with Fy of (82), there holds 


zeDe>ge R(F, YS ee” ldetdvenl - ee. (83) 


Here, 4; € C are the eigenvalues of the self-adjoint and positive compact operator 
Fy with corresponding normalized eigenfunctions yj € L?(S?). 


Historical Remarks and Open Questions 


Historically, the factorization method originated from the linear sampling method 
which will be explained in section “The Linear Sampling Method” (see also 
section “The Linear Sampling Method” in chapter > Inverse Scattering). The linear 
sampling method studies the far field equation Fg = 4, in contrast to the 
factorization method which characterizes the domain D by exactly those points z for 
which the modified far field equation an ie g = ¢- is solvable where Fy = (F* F)!/? 
in the case of section “Inverse Acoustic Scattering by a Sound-Soft Obstacle” and 
Fy = |Re F| + Im F in the case of section “Inverse Electromagnetic Scattering by 
an Inhomogeneous Medium.” It is easily seen that the points for which the far field 
equation F'g = ¢, is solvable determines only a subset of D — which can consist of 
a single point only, as the example of a ball shows. 

The implementation of the factorization method is as simple and universal as of 
the linear sampling method. Only the far field operator F —i.e., in practice a finite- 
dimensional approximation — has to be known. No other a priori information on the 
unknown domain D such as the number of components or the kind of boundary 
condition has to be known in advance. The mathematical justification, however, 
has to be proven for every single situation. Since their first presentations, the 
factorization method has been justified for several problems in inverse acoustic and 
electromagnetic scattering theory such as the scattering by inhomogeneous media 
[67,69, 72,73], scattering by periodic structures [11, 12], and scattering by obstacles 
under different kinds of boundary conditions [49, 73]. The factorization method can 
also be adapted for scattering problems for a crack [74] with certain modifications; 
we refer to the remarks concerning the crack problem in section “Other Generaliza- 
tions.” The factorization method for elastic scattering problems and wave guides is 
studied in [9] and [30], respectively. 

In many situations near field measurements on some surface I’ for point sources 
on the same surface I as incident fields rather than far field measurements for plane 
waves as incident fields are available. The corresponding “near field operator” M_ : 
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L?(C) — L?(P) allows a factorization in the form M = BGB’ where B’ is the 
adjoint with respect to the bilinear form jf, uvds rather than the (sesquilinear) inner 
product ie uvds. The validity of the range identity for these kinds of factorizations is 
not known so far and is one of the open problems in this field. For certain situations 
(see [73]), the corresponding far field operator F can be computed from M, and the 
factorization method can then be applied to F. 

Also the cases where the background medium is more complicated than the free 
space can be treated (see [48,73] for scattering problems in a half space and [71] for 
scattering problems in layered media). 

The justification of the factorization method for arbitrary elliptic boundary value 
problems or even more general problems is treated in [44, 70, 82]. 


4 Related Sampling Methods 


This section is devoted to some alternate examples of sampling methods which were 
developed during the last decade: the linear sampling method, first introduced by 
Colton and Kirsch in [35], the closely related MUSIC, the singular sources method 
by Potthast (see [85]), and Ikehata’s probe method (see [62]). However, it is not the 
aim of this section to report on all sampling methods. In particular, we do not discuss 
the enclosure method or the no-response test but refer to the monograph [86] and 
the survey article [87]. 


The Linear Sampling Method 


Here we reconsider the inverse scattering problem for time-harmonic plane acoustic 
waves of section “Inverse Acoustic Scattering by a Sound-Soft Obstacle,” i.e., 
the problem to determine the shape of an acoustically soft obstacle D from the 
knowledge of the far field pattern u(x; 6) for all ,6 € S. We refer to (42)- 
(47) for the mathematical model and the definition of the far field operator F from 
L?(S”) into itself. 

The factorization method for inverse scattering problems studies solvability of 
the equation Fig = ¢, in L*(S”) where Fy = (F*F)'/* in the case where 
F is normal (as, e.g., in section “Inverse Acoustic Scattering by a Sound-Soft 
Obstacle”) and Fy = | Re F| + ImF in the general case with absorption (see 
Theorems 14 and 18, respectively). In contrast to this equation, the linear sampling 
method considers the far field equation 


Fg=@, in L?(S?). (84) 


We mention again that in general no solution of this equation exists. However, 
one can compute “approximate solutions” g = g.. of (84) such that || g || 2:52) 
behaves differently for z being inside or outside of D. We refer to chapter > Inverse 
Scattering, Theorem 5.3, for a more precise formulation of this behavior. 
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The drawback of this result — and all the other attempts to justify the linear 
sampling method rigorously — is that there is no guarantee that the solution of a 
regularized version of (84), e.g., by Tikhonov regularization, will actually pick the 
density g = g,, with the properties of the aforementioned “approximate solution.” 
We refer to [53] for a discussion of this fact. However, numerically the method has 
proven to be very effective for a large class of inverse scattering problems (see, e.g., 
[26] for the scattering by cracks, [27] for inverse scattering problems for anisotropic 
media, [19] for wave guide scattering problems, [33, 34, 51] for electromagnetic 
scattering problems, and [29, 31,40] for elastic scattering problems). Modifications 
of the linear sampling method and combinations with other methods can be found 
in [8, 20, 79]. 

For the cases in which the factorization method in the form (F* F)!/ ae = 
¢, is applicable, a complete characterization of the unknown obstacle D by a 
modification of the linear sampling method can be derived by replacing the indicator 
value |lgl| ,2¢52) by (g, @;)12(s2). This is summarized in the following theorem (see 
(10, 13] and, for the following presentation, [73]). 


Theorem 19. Let u® = u(x; 6) be the far field pattern corresponding to the 
scattering problem (42)-(45) with associated far field operator F, and assume that 
k? is not a Dirichlet eigenvalue of —A in D. Furthermore, for every z € D, let 
g, € L*(S?) denote the solution of (F* F)'/4g, = @,, i.e., the solution obtained by 
the factorization method, and for every z € R° and ¢ > 0, let g = gz € L?(S7) be 
the Tikhonov approximation of (84), i.e., the unique solution of 


(el + F*F)g = F*¢, (85) 


which is computed by the linear sampling method (if Tikhonov’s regularization 
technique is chosen). Here, ¢, € L*(S7) is defined in (55). Furthermore, let 
Vg. e(Z) = (S26, ¢) 1282) = fo2 g-(0) exp(ikd . z)ds(9) denote the corresponding 
Herglotz wave function evaluated at z. 


(a) For every z € D, the limit lime-+o Vg, , (z) exists. Furthermore, there exists c > 
0, depending on F only, such that for all z € D the following estimates hold: 


2 : 2 
c Il gzllZ2(s2) = lim |v2..(2)| = I|SzllZ2(s2) - (86) 


(b) Forz ¢ D, the absolute values |v, .(z)| tend to infinity as € tends to zero. 


Proof. Using an orthonormal system {y; : j € N} of eigenfunctions w; corre- 
sponding to eigenvalues A; € C of F, one computes the Tikhonov approximation 
&~¢ from (85) as 


630 M. Hanke-Bourgeois and A. Kirsch 


ee Bg es Wid2682))- 


j=! 


From ve(z) = (g, $:)12(s2) for any g € L?(S”), we conclude that 


Sap 
Ve.e(2) = > 4 — |b. WA) (82 


87 
2. jee (87) 
j=l 
(a) Let now z € OD. Then (F*F)'/4g, = @, is solvable in L*(S*) 
by Theorem 14, and thus (,,Wj)z2(52) = ((F*F)"4.,Wj)1(52) = 
(82, (F* F)*W)12(92) = VIA s|(@e Ws) 2218: 
Therefore, we can express vg. ,(z) as 
jlA;\ AG IASI 
Vel) = LU inP +e Beg Ke Waders)" = Me: lia» LOT Ee aj e’ 
(88) 


where p; = (ges Wy )z292)/ /WWeellz2 2(s2) 18 nonnegative with Xj p; = 1. An 
elementary argument (theorem of dominated convergence) yields convergence 


ails 
es: > Lon = Lam 


j=l j=l 


as € tends to zero where again s; = A;/|A;|. The properties of p; imply that 
the limit belongs to the closure C of the convex hull of the complex numbers 
{s; : 7 © N}. The same argument as in the proof of Theorem 13 yields that C 
has a positive distance c from the origin, i.e., pen pisy| > c which proves 
the lower bound. The upper estimate is seen directly from (88). 

(b) Let now z ¢ D, and assume on the contrary that there exists a sequence {¢,,} 
which tends to zero and such that |v, (z)| is bounded. Here we have set v, = 
vg.,, for abbreviation. Since s; converges to 1, there exists jo € N with Red; > 
0 for j => jo. From (87) for ¢ = &,, we get 


an te Aj 
Vn(Z) = dX eee (¢:. Wi) 189)" + sn Pte \(¢:. Vi) 12(89)| - 


J=Jo 


Since the finite sum is certainly bounded for 1 € N, there exists c) > 0 such 
that 


Aj 
> pepe |e wy i99 <c, foralln € N. 


J=Jo 


Sampling Methods 631 


Observing that for any complex number w € C with Rew > 0 and Imw > 0 
we have that Re w + _Imw > |w|, we conclude (note that also ImA; > 0) 


~~. Red; +Im/; 
> yo — s+ 


dj 
2c, <2 3 De (eae, |(¢:, Viiysyl| = / bes Wi )z254] 


(2 
J=Jo j=io |A;| ae En 
ye ay ; 
> ——— 7 - Ais 
—_ Xu |A; |? + & l(b: Wiisy) = > aly, JA; |? + &n + Ep \(¢: WAL (s2)| 


for alln € N andall J > jo. Letting n tend to infinity yields boundedness 
of the finite sum uniformly w.r.t. J and thus convergence of the series 
eae Tol \(¢:. Wi) eal From (62) therefore follows that z € D, which 
is the desired contradiction. a 


Obviously, this kind of modification of the original linear sampling method can 
be done for all inverse scattering problems for which Theorem 14 holds. This 
includes scattering by acoustically hard obstacles or inhomogeneous nonabsorbing 
media or, with appropriate modifications, scattering by open arcs. 


MUSIC 


The linear sampling method investigates “to what extent” the far field equation 


Fg = ¢, 


is solvable for a number of sampling points z within some region of interest. As 
we have mentioned before, this equation has a solution in very rare cases only and 
usually not for every z € D. 

However, if the obstacle is very small, then it turns out that the far field operator 
almost degenerates to a finite rank operator, in which case the “numerical range” of 
F and (F* F)!/4 would be the same finite-dimensional subspace, where the latter 
is known to contain ¢, for every z € D — under appropriate assumptions on the 
particular problem setting (see Sects. 2 and 3). 

To investigate this observation in more detail, we embed the real scene in a 
parameterized family of problems, where the parameter 5 > 0 reflects the scale 
of the problem. Assume that the scatterer D = U?_, Dj; consists of m obstacles 
given as 


D; = z% + 6U; B= [gn ecg Mj (89) 


where each domain U; contains the origin and has Lipschitz continuous boundary 
and the closure of U; has a connected complement. We shall call z; the location 
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of D; and U; its shape. We focus our presentation on an inhomogeneous medium 
setting for acoustic scattering, i.e., the Helmholtz equation, to provide analogies to 
both settings from Sect.3. Let po and co be the density and the speed of sound in 
vacuum, k = w/co be the associated wave number with frequency w, and u' (x) = 
exp (ikx- 6) be an incoming plane wave. Then, if we assume that the density p; and 
the sound of speed c; in each object D; are real and constant, then the total field 
uj =u t+ ux Solves the Helmholtz equation (see, e.g., [36]) 


1 
div (- grad ws) + onus =0 in R’, (90) 


with the radiation condition 


dus +78 —2 
ar —ikuy = O(r~*) forr = |x| > ~, (91) 
uniformly with respect to X = x/|x|, and the parameter 7 equals no = 1/0 in 


R3\D, and n; = c)/(c?p;) in D;, i = 1,...,m, respectively. We mention that for 
constant 7 = 1/o, it has been shown in [72] that the standard factorization method 
(with Fy = (F*F)"/?) applies for this setting with fixed scaling parameter 5. We 
know of no result, however, where the factorization method is used to reconstruct 
the supports of p — po and 7 — no in this setting simultaneously, although there are 
partial results for a similar problem (in a bounded domain and with a different sign 
of 7) arising in optical tomography, cf. [43,59]. 

The idea to approach this problem is based on an asymptotic expansion of the far 
field us° of the scattered wave with respect to the parameter 6 in (89). We quote the 
following result from [4]. 


Theorem 20. The far field of the scattering problem (90) and (91) for the scatterers 
given in (89) satisfies 


uS°(£; 6) 
= 3) ((4 = i) &-Mi6 -— (2 - 1) ul) exp (ik(6 — 2) -z;) + 0(8), 
jay + AO “0 
(92) 
and the associated far field operator can be rewritten as 
F = 58°F +0(8°) (93) 


in the norm of L(L7(S*)), where the rank of the operator F is at most 4m. Here, 
|U;| is the Lebesgue measure of U;, and M; € R°*? are symmetric positive definite 
matrices that depend on the shape U;, the so-called polarization tensors. 
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As is obvious, the scattered field and its far field vanish as 6 — O. The 
corresponding rate 5° reflects the space dimension; in R*, the corresponding field 
decays like 5? as 6 > 0. 

The importance of Theorem 20 stems from the fact that the leading order 
approximation F of the far field operator F has finite rank, whereas F' has infinite 
dimensional range. The rank of F is 4m, unless some of the scatterers have the same 
material parameters as the background vacuum. Note that the dominating term of 
us° consists of two parts: the first contribution stems from the change in the density 
p and corresponds to the far field of a dipole (point source) in z;; likewise, the second 
term corresponds to the far field of a monopole in z;, and this is the result of a change 
in the parameter 7. 

It is easy to deduce from Theorem 20 that we can factorize F quite naturally in 
three factors. 


Theorem 21. The operator F: L?(S*) — L?(S?) admits a factorization of the 
form 


F = —BMB’, (94) 


where B : C*” -+ L*(S*) maps a vector [pi,..., Pms Q1s-.+,dm]’ € C*” with 
pi € C’ anda; € C, i = 1,...,m, to the far field of 


m 


u(x) = x (pi - grad, P(x, z;) + aj B(x, z)), 


i=1 


where ® is as in(51), M € R*”"**" is a real block diagonal matrix with m blocks of 
size 3 x 3 and single elements on its diagonal, and M is nonsingular, if and only 
if 0; 4 po and n; # no for alli = 1,...,m. The operator B' is the dual operator 
of B with respect to the bilinear forms of C*" and L*(S*), i.e., B'g consists of the 
gradients and point values of the Herglotz wave function 


ve(x) =f g(0) exp(ikx-0)ds(6), xe R’, 


evaluated at the points z;, i = 1,...,m. 


As M in (94) is invertible, the range of F and the range of B coincide, and it 
consists of the far fields of the monopoles and all possible dipoles emanating from 
the locations z; of Dj, i = 1,..., m. Using the unique continuation principle, we 
can thus conclude the following result. 


Corollary 2. If each scatterer has a different parameter n than the background 
medium, then a point z € R? is the location z; of one of the scatterers, if and only if 
o, of (55) belongs to the range of F. 
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When 6 is small, it follows from (93) that numerically the range of F and the 
range of F are the same, essentially. By this we mean that the dominating 4m 
singular values of F are small perturbations of the nonzero singular values of FE, 
and the corresponding singular subspaces are also close to each other. Moreover, 
we expect to see a sharp gap between the 4mth and the 4m + Ist singular value 
of F. We can search for this gap to determine the number m of the scatterers 
and then determine the angle between the test function ¢, and the 4m-dimensional 
dominating singular subspace of F'. When z is close to the location of one of the 
scatterers, then this angle will be small, otherwise this angle will be larger. This 
way images can be produced that enable one to visualize the approximate locations 
of the scatterers, but not their shape. 

This approach applies for all problem settings that have been discussed in Sects. 2 
and 3 and many more. In impedance tomography, for example, the corresponding 
asymptotic expansion of the boundary potential has the form 


m 


us(x) — uy(x) = 5" D> 


i=1 


= grad. N(x, z;)- Mj grad uy (z;) + 0(6"), x € AQ 


(95) 


where n is again the space dimension, N the Neumann function (7), and M; 
the associated polarization tensor, cf. [28] or section “Asymptotic Analysis of 
the Voltage Perturbations” in chapter » Expansion Methods. The leading order 
approximation of the difference between the associated Neumann-Dirichlet oper- 
ators, As — Ay, can be factorized in a similar way as in Theorem 21 and has an 
nm-dimensional range that is spanned by dipole potentials sitting in the locations z; 
of the obstacles D;, i = 1,...,m; recall that 1 is the space dimension. 

For the full Maxwell’s equations considered in section “Inverse Electromagnetic 
Scattering by an Inhomogeneous Medium,” the range space of the corresponding far 
field operator F' of (77) consists of the magnetic far fields corresponding to electric 
dipoles at the infinitesimal scatterers; if the scatterers also differ in their magnetic 
permeability, then the range space also contains the far fields of the magnetic dipoles 
inz,i=1,...,m. 

The method described above for reconstructing the locations of small scatterers 
is often called MUSIC in the inverse problems community. Originally, the MUSIC 
algorithm is a signal processing tool for frequency estimation from the noisy 
spectrum of some signal (MUSIC stands for MUltiple SIgnal Classification.), cf., 
e.g., [90]. In a seminal report (Devaney, Super-resolution processing of multi-static 
data using time reversal and MUSIC. Unpublished manuscript, 2000), this algorithm 
was suggested to detect “point scatterers” on the basis of the Born approximation, 
which led to an algorithm that is not exactly the same, but related to the one we 
have sketched above. The relation between this algorithm and the factorization 
method has subsequently been recognized in [32, 69]. However, although the form 
of the factorization (94) is similar to the ones for the factorization method derived 
in Sects. 2 and 3, it is slightly different in its precise interpretation; this has been 
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exemplified in [2] by taking the limit of each of the factors from Theorem 4 as 
5 > 0. 

The derivation of asymptotic formulas as in Theorem 20 goes back to the 
landmark paper [41]. In [24], formula (95) from [28] was used to provide the 
rigorous foundation of the MUSIC-type algorithm from above. Important extensions 
and generalizations to other problem settings include [1, 4, 7, 46, 91]; for a more 
detailed survey and further references, we refer to chapter » Expansion Methods 
and the monographs [5, 6]. 

Numerical illustrations of this approach can be found in various papers (see, e.g., 
[3, 24, 46]). 


The Singular Sources Method 


As in section “Inverse Acoustic Scattering by a Sound-Soft Obstacle,” we reconsider 
the simple inverse scattering problem for the Helmholtz equation in R* to determine 
the shape of an acoustically soft obstacle D from the knowledge of the far field 
pattern u(x; 6) for all x, 6 € S?. We refer again to (42)-(47) for the mathematical 
model and the definition of the far field operator F from L?(S7) into itself. Note 
that again wu = u(x; 6) and u® = u(x; 6) denote the scattered field and far field 
pattern, respectively, corresponding to the incident plane wave of direction 6 S?. 
The basic tool in the singular sources method is to consider also the scattered 
field v’ = v°(x; z) which corresponds to the incident field v(x) = ®(x, z) of (51) 
of a point source, where z ¢ D is a given point. The scattered field v’(z; z) evaluated 
at the source point blows up when z tends to a boundary point. One can prove (see 
[73, 86]) that there exists a constant c > 0 (depending on D and k only) such that 


|v°(z; for all z ¢ D. (96) 


z)| > ee laa 
d(z, 0D) 


Here, d(z, dD) = inf {|z—y| : y € dD} denotes the distance of z to the boundary 
of D. 

The idea of the singular sources method is to fix z ¢ D and ¢ > 0 and a bounded 
domain G, C R?® such that its exterior is connected and z ¢ G, and D C G,. 
Runge’s approximation theorem (see, e.g., [73]) yields the existence of g € L?(S?) 
depending on z, G, and € such that 


lve- ®C-. Deg = & (97) 


where v, denotes the Herglotz wave function, defined by 


v(x) = 2 (0) exp(ikx-@)ds(0), xe R?. 
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In the following, only the dependence on ¢ is indicated by writing g,. The following 
convergence result for the Singular Sources Method is known (see [73, 86]). 


Theorem 22. Let u = u%(£:0), £8 € S? be the far field pattern of the 
scattering problem (43-45). Fix z ¢ D and a bounded domain G, C R°? such 
that its exterior is connected and z ¢ G, and D C G;. For any € > 0, choose 
g = g. € L?(S*) with (97). Then 


lim lim [ (Fg,)(—6)¢5(6)ds(6) = v*(z:2), 


5—>0e>0 


i.e., by substituting the form of F, 


lim lim [ [ _u(6: f)ge(Agu Ods(Hds(G) = v2). 


5>0e>0 


Note that the limits are iterated, 1.e., first the limit w.r.t. ¢ has to be taken and then 
the limit w.r.t. 6. 
Combining this result with (96) yields 


lim lim 
560 e>0 


[. [or ceineres@asnas@)|> Sp 8 


This result assures that for z sufficiently close to the boundary dD (and regions G, 
chosen appropriately) the quantity 


lim lim 
b6>0e>0 


in uP (0: feces Bas(Hpas(A) 
g2 J 52 


becomes large. 
It is convenient to use domains G, of the special form 


Gap = +00) +} x Rs ix] =, ~ p> —cosph 


for some (large) radius R > 0, opening angle 6 € [0, 2/2), direction of opening 
p € S?, and p > 0. The dependence on £, p, and R is not indicated since they are 
kept fixed. This domain G,,, is a ball centered at z + pp with radius R from which 
the cone of direction —p and opening angle 6 have been removed. Obviously, it 
is chosen such that z ¢ G,. These sets G,p are translations and rotations of the 
reference set 


G=)xeR : [al < R77 p> cop 
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for p = (0,0, 1)", ie., G., =z+ MG for some orthogonal M € R?*?. 

With these transformations, we can consider the singular sources method as a 
sampling method with sampling objects z and M. 

From the arguments used in the proof of Theorem 22, it is not clear whether or 
not the common limit lim, 5.9 exists. However, if k? is not a Dirichlet eigenvalue 
of —A in D, then the following stronger result than (49) can be obtained by using 
the factorization (49). 


Theorem 23. Let z ¢ D and G, C R? be a bounded domain such that its exterior 


is connected and z ¢ G, and D C G,. For any ¢ > 0, choose g, € L?(S7) with (97) 
with respect to the H!' —norm, i.e., 


Ye. -— PC.) lane =e 


Assume furthermore that k? is not a Dirichlet eigenvalue of —A in D. Then there 
exists a constant c > 0 depending only on D and k such that 


tim | f°: fees Bas Ads(B)) = ies (Pee. 8) 1248 


c 
> —__.. 
~ d(z, dD) 


For a proof we refer to [73]. Numerical reconstructions with the Singular Sources 
Methods are shown in [86]. 


The Probe Method 


The probe method has originally been proposed in [62] for the inverse problem of 
impedance tomography of section “Conducting Obstacles,” and here we also restrict 
our attention to this setting. To be precise, let o € L®(&2) be a (complex valued) 
admittivity function, and define u € H. (2) as the unique (weak) solution of the 
boundary value problem 


a 
div(ogradu) =O inQ oa =f ondQ / uds = 0, (99) 
v aQ 


where f € LZ (dQ). For the spaces HY(Q) and LZ(6Q), we refer to sec- 
tion “Impedance Tomography in the Presence of Insulating Inclusions.” 

As in section “Conducting Obstacles,’ we assume that o € L™°(Q2) is a 
perturbation of the constant background admittivity function og = 1. More 
precisely, let D C Q be again the finite union of domains such that Q\D is 
connected and o = 1 in Q\D, and let there be a constant co > O such that 


Imo(x) <0 and Reo(x) > 1+cp on D. (100) 
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The case 0 < cg < Re o(x) < 1 —co can be treated in a similar way (see [73]). 
The unique solvability of the direct problem, i.e., the boundary value problem (99), 
guarantees existence of the Neumann-to-Dirichlet operators A, Aq : LZ (0Q) > 
L3 (8) corresponding to o and oy = 1, respectively. 

As in section “Conducting Obstacles,” the goal of the inverse problem is to 
determine the support D of o — | from the knowledge of the absolute data A or the 
relative data A— A. The difference to the setting in section “Conducting Obstacles” 
is that o is now a scalar and complex-valued function. 

In the probe method, the sampling objects are curves in Q starting at the 
boundary dQ of Q. In the original paper [62], these curves are called needles. We 
keep this notation but mention that — perhaps in contrast to the colloquial meaning 
— these needles do not need to be straight segments but can be curved in general. 
By choosing a family of needles, the probe method determines the first point on 
the needle which intersects the boundary dD (see Theorem 24 below). Therefore, 
in contrast to the factorization method and the linear sampling method, the probe 
method tests on curves instead of points. 


Definition 1. A needle C is the image of a continuously differentiable function 7 : 
[0, 1] + Q such that n(0) € dQ and n(t) € Q for all t € (0, 1] and n/(t) 4 0 for 
allt € [0, 1] and n(t) 4 n(s) fort 4s. We call n a parameterization of the needle. 
The following monotonicity property is the basic ingredient for the Probe 
Method. 
Under the above assumptions on a0 € L®(Q), there exists c > 1 such that 


1 
~| |grad uy|7dx < Re(f, (Ay — A) f) <c| |grad wy |*dx (101) 
CJD D 


for every f € LZ (dQ). Here, uy € H. 9(2) denotes the unique solution of (99) for 
the constant background case oy = I. 

Let 7 : [0, 1] — Q be the parameterization of a given needle, t € (0, 1] a fixed 
parameter, and C,; = {n(s) : 0 <s < ¢} the part of the needle from s = Oto s = ¢. 
Let ®(x, y) denote the fundamental solution of the Laplace equation, e.g., 


&(x, y) = ———— 
(x,y) ial 


in R?. Runge’s approximation theorem (see, e.g., [73]) yields the existence of a 
sequence w, € H'(Q) of harmonic functions in Q such that 


I Wn = D(., n(t)) lay 0, n> oo (102) 
for every open subset U with U C Q\C,. We set f, = dw,/dv on dQ and note 


that f, depends on C, but not on the unknown domain D. The dependence on C; is 
denoted by writing 7, (C;). It can — at least in principle — be computed beforehand. 
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Theorem 24. Let the above assumptions on o hold and fix a needle with parame- 
terization n : [0,1] > Q. Define the set T C [0, 1] by 


T=)te€ (0,1): sup {IRe( fn(Ci), (A — Ax) fn(Cr))| < 00}. (103) 


ne 


Here, f,(C;) = dw,/dv € H~'/2(9Q) is determined from (102) (so far, we have 
chosen the boundary current f in (99) from i (OQ) for convenience; however, 
the quadratic form in (103) extends as dual pairing (H~'/*(0Q), H'/*(9Q)) to 
f € H~'/?(0Q) with vanishing mean). Then 7 4 Q, and one can define t* = 
sup {t € [0, 1] : [0,¢] € 7}, which satisfies 


ita min {t € [0, 1]: n(t) € OD}, ifC; N D 
= D 


ZO 
1, ifC; NDAD. a0") 


We recall that C; = C = {n(t) : t € [0, 1]}. 

For a proof we refer to [62,73]. 

Note that for every needle the set 7 of the form (103) is determined by the given 
data: it depends on 7 and the approximating functions w,,. Formula (104) provides 
a constructive way to determine 0D from A — Aj: one has to choose a family 
of needles which cover the domain Q, and for each needle one computes ¢* as 
the largest point of 7; if t* < 1, then n(t*) € dD. Obviously, this procedure is 
very expensive from a computational point of view. However, if one samples with 
“linear” needles only, i.e., rays of the form C = {z+ tp: t > 0}: Q for z € Q and 
unit vectors p € S7, then the computational effort can be reduced considerably since 
the approximating sequence (102) has to be computed only once for a reference 
needle. However, by using only rays as needles, one can not expect to detect the 
boundary of D completely. Only the “visible points” of dD can be detected, i.e., 
those which can be connected completely in Q\D by straight lines to dQ. 

In an implementation of the definition of 7 of (103), one has to decide whether 
a supremum is finite or infinite. Numerically, this is certainly not an easy task. In 
[62], it has been suggested to replace 7 of (103) by 


Tu = {t € [0,1]: sap (Ret snl@2) — Aa) fn(Gi))| < M3} 


for some M > 0, for which a result analogously to the one in Theorem 24 can be 
established. We refer to [62] for more details. 

Again, the probe method is general enough to have extensions to a number of 
related inverse problems in elasticity (see [63]) and scattering theory (see [61]). For 
numerical reconstructions, we refer to [87]. 
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5 Conclusion 


This chapter has considered a very general approach to identify obstacles within 
a homogeneous background. The method requires the underlying physics to be 
described by an elliptic differential equation and utilizes the fact that signals become 
smoother the longer they travel through the homogeneous material. 

This technique has been exemplified for two model problems from impedance 
tomography and inverse scattering. Relations of this method to other techniques 
such as the probe method and MUSIC have also been explored. 


6 Appendix 


In this appendix, we collect some functional analytic results on range identities. 
The factorization method makes use of the fact that the unknown domain D can be 
characterized by the range of some compact operator A : X — Y where A is 
related to the known operator M : Y — Y through the factorization 


M =AGA*. (105) 


Throughout this whole chapter, we assume that Y is a Hilbert space and X a 
reflexive Banach space with dual X*. We denote by A* : Y — X™* the adjoint of 
A, where Y is identified with its dual. 

For a computable characterization of D, the range of the operator A has to be 
expressed by the operator M which is the goal of the range identity. 

In the simplest case where also X is a Hilbert space and G is the identity 7, the 
range identity is easily obtained via the singular system of A and the Theorem of 
Picard. We recall that {o;, x;, yj; : j € J} is a singular system of a linear and 
compact operator T : X — Y between Hilbert spaces X and Y if {x; : j € J} 
and {y; : j € J} are complete countable orthonormal systems in the subspaces 
N(T)+ c X and N(T*)+ C Y, respectively, and oj; € Ryo such that Tx; = 0; y; 
and 7* y; = 0;x; forall j € J. 

We note that {o?. xjPise J}, together with a basis of the null-space NV’(T) of 


T and associated eigenvalue 0, is an eigensystem of the self-adjoint and nonnegative 
operator 7*7 . Furthermore, 


Tx =) loj(x.x)x yj, xe XxX, 
jel 

T*y =) oj(y, yj vxj, yeY 
jel 


Theorem 25 (Picard). Let X, Y be Hilbert spacesandT : X — Y be acompact 
operator with singular system {o;, X;, yj : j € J}. Then there holds: An element 
y €Y belongs to the range R(T) of T, if and only if, 
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\o 2 
y EN(T*)+ and - 1 ydel <0o 
jes J 


For a proof we refer to, e.g., [39]. Applying this theorem to the factorization (105) 
with G = I, and when X* is identified with X, one obtains. 


Corollary 3. Let A: X — Y beacompact operator between Hilbert spaces X and 
Y with dense range and M = AA* : Y — Y. Then the ranges of A and M'/? coin- 
cide. Here, the self-adjoint and nonnegative operator M'/? : Y — Y is given by 


MPy=> VAO.y)vyy, VEY, 


jel 
where {y; : j € J} are the orthonormal eigenelements of the self-adjoint, compact, 


and nonnegative operator M corresponding to the positive eigenvalues i;. It 
follows that 


2 
yEeR(A) => zea < 00 
7 


jeJ 


For more general factorizations of the form M = AGA%, the following (preliminary) 
characterization is useful (see [68]; for an equivalent formulation, see Theorem 3 
of [82]). 
Theorem 26. Let X be a reflexive Banach space with dual X* and dual form (-, -) 
in (X*, X). Furthermore, let Y be a Hilbert space and M : Y > Y and A 
X — Y be linear-bounded operators such that the factorization (105) holds for 
some linear and bounded operator G : X* — X, which satisfies a coercivity 
condition of the form: There exists c > 0 with 
Ke, Ge)| = cllellx= forallg € R(A*) Cc X*. (106) 
Then, for any ¢ € Yd £0, 
gE R(A) => inf {\(W Mv)y| : WEY, é)y = 1 > 0. (107) 
Proof. The form |(w, Mw)y| can be estimated by 
lv. Mp)y| = |(A*¥,GA*W)| > cl|A*Wly= forall ye Y. (108) 


Let first 6 = Ago for some go € X. For w € Y with (w, d)y = 1, there holds that 
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Cc 
lu. MW)y| > cllA*vllx« = Tool A* Vl ll gollx 
OLX 
Cc Fe (6 2 Cc 
> —>|(A*¥, go) |? = —|(v, Ago )y| oak aT ae 
llvolly lolx llyo ll 


This provides the lower bound of the infimum. 

Second, assume that 6 ¢ R(A). Define the closed subspace V := {yr € Y 
(vw, @)y = O}. Then A* (V) is dense in R(A*) C X™*. Indeed, this is equivalent 
to the statement that the annihilators [A*(V)]+ and [R(A*)]+ = N(A) coincide. 
Therefore, let y € [A*(V)]+, ie., (A*W, v) = Oforall y € Vyie., (W, Ag)y = 0 
for all wy € V,ie., Ap € Vt = span {$}. Since ¢ ¢ R(A), this implies Ag = 0, 
ie. g € N(A). Therefore, A*(V) is dense in R(A*). 


Choose a sequence {w,} in V such that A* yy, — ——1, A*¢ as n tends to infinity 


~ Welly 
and set Ww = Wnt+¢/ ale Then (W,, )y = land A*w, — 0. The first equation 
of (108) yields 


(Wn, MVn)v| < (GI A* Walla 


and thus (Wy), Mw,)y — 0,1 > oo, which proves that inf {|(W, Mw)y|: w € Y, 
(W.o)y = 1} =0. 

We note that the inf-condition only depends on M and not on the factorization. 
Therefore, we have as a corollary. 


Corollary 4. Let Y be a Hilbert space and X, and X2 be reflexive Banach spaces 
with duals X* and X3, respectively. Furthermore, let M : Y — Y have two 
factorizations of the form M = A, G,Af = Az G2A5 as in (105) with compact 
operators A; : Xj; — Y and bounded operators Gj : X; —> Xj, which both 
satisfy the coercivity condition (106). Then the ranges of A, and A> coincide. 


Corollary 4 is useful for the analysis of the factorization method as long as M is 
normal. However, there are many scattering problems for which the corresponding 
far field operator fails to be normal, e.g., in the case of absorbing media. For these 
problems, one can utilize the self-adjoint operator 


My = |Re M| +ImM, (109) 


which can be computed from M. Note that Re M = +(M + M*) andIm M = 
+(M — M*) are again self-adjoint and compact, and the absolute value |Re M| of 
Re M is defined to be 


IReMly =D Alvar. wey, 


jes 


where {A;, y; : j € J} denotes the spectral system of Re M. 
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Now we can apply Corollary 4 to obtain the following result (see [73] for the 
lengthy proof and [76] for a weaker form of assumption (d)). 


Theorem 27. Let X be a reflexive Banach space with dual X* and dual form 
(-, +) in (X*, X). Furthermore, let Y be a Hilbert space and M : Y —> Y and 
A: X — Y be linear-bounded operators such that the factorization (105) holds 
true for some linear and bounded operator G : X* —> X. Furthermore, let the 
following conditions be satisfied: 


(a) The range of A is dense in Y. 

(b) There holds Re G = Go + Gi, where Go satisfies (106) and G, : X* —> X is 
compact. 

(c) The imaginary part Im G of G is non-negative, i.e., Im(y, Gy) > 0 for all 
ge X*, 

(d) G is injective or Im G is positive on the null-space of Re G. 


Then the self-adjoint operator My of (109) is positive, and the ranges of A and 


My,” 


coincide. 
As an immediate corollary, we have: 
Corollary 5. Let M: Y > Y and A: X > Y andG: X* => X be as in 


Theorem 27, and let G be self-adjoint, i.e, G* = G, and satisfy (106). Then the 
ranges of A and M\/? coincide, and 


+) 42 
ye R(A = pa < oO 
jel J 


where {A;, yj : J € Jj denotes a spectral system of the self-adjoint and compact 
operator M = AGA*. 
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Abstract 


We give a survey of the mathematical basis of inverse scattering theory, con- 
centrating on the case of time-harmonic acoustic waves. After an introduction 
and historical remarks, we give an outline of the direct scattering problem. 
This is then followed by sections on uniqueness results in inverse scattering 
theory and iterative and decomposition methods to reconstruct the shape and 
material properties of the scattering object. We conclude by discussing qualitative 
methods in inverse scattering theory, in particular the linear sampling method and 
its use in obtaining lower bounds on the constitutive parameters of the scattering 
object. 


1 Introduction 


Scattering theory is concerned with the effects that obstacles and inhomogeneities 
have on the propagation of waves and in particular time-harmonic waves. In the 
context of this book, scattering theory provides the mathematical tools for imaging 
via acoustic and electromagnetic waves with applications to such fields as radar, 
sonar, geophysics, medical imaging, and nondestructive testing. 

For reasons of brevity, in this survey, we focus our attention on the case of 
acoustic waves and only give passing references to the case of electromagnetic 
waves. We will furthermore give few proofs, referring the reader interested in further 
details to [22]. Since the literature in the area is enormous, we have only referenced 
a limited number of papers and hope that the reader can use these as starting point 
for further investigations. 

Mathematical acoustics begins with the modeling of acoustic waves, i.e., sound 
waves. The two main media for the propagation and scattering of sound waves are 
air and water (underwater acoustics). A third important medium with properties 
close to those of water is the human body, i.e., biological tissue (ultrasound). Since 
sound waves are considered as small perturbations in a gas or a fluid, the equation 
of acoustics, i.e., the wave equation 


P 
2K 1 
2 972 P (1) 


for the pressure p = p(x,t), is obtained by linearization of the equations for the 


motion of fluids. Here, c = c(x) denotes the local speed of sound, and the fluid 
velocity is proportional to grad p. For time-harmonic acoustic waves of the form 


p(x,t) = Re {u(x) e"@"! (2) 
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with frequency w > 0, it follows that the complex-valued space-dependent part u 
satisfies the reduced wave equation 


ie 
Au+ —ju=0. (3) 
c 


Here we emphasize that the physical quantity describing the sound wave is the real- 
valued sound pressure p(x,t) and not the complex-valued amplitude u(x) in the 
representation u(x) e~'®’. For a homogeneous medium, the speed of sound c is 
constant and (3) becomes the Helmholtz equation 


Au+k*u=0, (4) 


where the wave number k is given by the positive constant k = w/c. 
A solution to the Helmholtz equation whose domain of definition contains the 
exterior of some sphere is called radiating if it satisfies the Sommerfeld radiation 


condition 
lim r (= - ik’) = 0, (5) 
or 


where r = |x| and the limit holds uniformly in all directions x /|x|. Here, and in the 
sequel, |x| := ,/x7 +x} + x} denotes the Euclidean norm of x = (x1,.x2,%3) € 


IR>. For more details on the physical background of linear acoustic waves, the reader 
is referred to [66]. 

We will confine our presentation of scattering theory for time-harmonic acoustic 
waves to two basic problems, namely, scattering by a bounded impenetrable obstacle 
and scattering by a penetrable inhomogeneous medium of compact support. For a 
vector d € R? with |d| = 1, the function e’***@ satisfies the Helmholtz equation 
for all x € RR?. It is called a plane wave, since e'***4-®") is constant on the 
planes kx-d — wt = const. Note that these wave fronts travel with velocity 
c in the direction d. Assume that an incident field is given by the plane wave 
u(x) = e'**°¢. Then the simplest obstacle scattering problem is to find the 
scattered field u* as a radiating solution to the Helmholtz equation in the exterior 
of a bounded scatterer D such that the total field u = u' + u satisfies the Dirichlet 
boundary condition 


u=0O ondD (6) 


corresponding to a sound-soft obstacle with the total pressure, i.e., the excess 
pressure over the static pressure po, vanishing on the boundary. Concerning the 
geometry of scattering obstacles, for simplicity, we always will assume that D is 
a bounded domain with a connected boundary dD of class C?. In particular, this 
implies that the complement R?\ D is connected. However, our results remain valid 
for a finite number of scattering obstacles. 
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Boundary conditions other than the Dirichlet condition also need to be considered 
such as the Neumann or sound-hard boundary condition 


a =0 ondD (7) 
dv 


or, more generally, the impedance boundary condition 


Loe eee on oD, (8) 
dv 


where v is the outward unit normal to dD and A is a positive constant called 
the surface impedance. More generally the impedance A can also vary on oD. 
Since grad u is proportional to the fluid velocity, the impedance boundary condition 
describes obstacles for which the normal velocity of the fluid on the boundary 
is proportional to the excess pressure on the boundary. The Neumann condition 
corresponds to a vanishing normal velocity on the boundary. In order to avoid 
repetitions by considering all possible types of boundary conditions, we will in 
general confine ourselves to presenting the basic ideas in acoustic obstacle scattering 
for the case of a sound-soft obstacle. 

The simplest scattering problem for an inhomogeneous medium assumes that 
the speed of sound is constant outside a bounded domain D. Then the total field 
u=u' +! satisfies 


Au+tk?nu=0 inR}, (9) 


and the scattered wave u* fulfills the Sommerfeld radiation condition (5), where the 
wave number is given by k = w/co andn = ca v4 c? is the refractive index given by 
the ratio of the square of the sound speeds c = co in the homogeneous host medium 
and c = c(x) in the inhomogeneous medium. The refractive index is positive and 
satisfies n(x) = 1 for x ¢ D, and we assume 7 to be continuously differentiable in 
R3 (our results are also in general valid for n being merely piecewise continuous in 
IR>). An absorbing medium is modeled by adding an absorption term which leads to 
a refractive index with a positive imaginary part of the form 


2 
n= a +1 r 
in terms of a possibly space-dependent absorption coefficient y. 

Summarizing, given the incident wave and the physical properties of the scatterer, 
the direct scattering problem is to find the scattered wave and in particular its 
behavior at large distances from the scattering object, i.e., its far-field behavior. The 
inverse scattering problem takes this answer to the direct scattering problem as its 
starting point and asks what the nature of the scatterer that gave rise to such far-field 
behavior is. 
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To be more specific, it can be shown that radiating solutions u* to the Helmholtz 
equation have the asymptotic behavior 


ebk lx! . 1 
“(x)= JHo(® +0 (—) » |x| > 0, (10) 
|x| |x| 
uniformly for all directions X = x/|x|, where the function uo. defined on the 


unit sphere S? is known as the far-field pattern of the scattered wave. For plane 
wave incidence, we indicate the dependence of the far-field pattern on the incident 
direction d and the observation direction X by writing Ugo = Ugo(X, d). The inverse 
scattering problem can now be formulated as the problem of determining either 
the sound-soft obstacle D or the index of refraction n (and hence also D) from a 
knowledge of the far-field pattern uoo(%, d) for and d on the unit sphere S? (or a 
subset of 5”). 

One of the earliest mathematical results in inverse scattering theory was Schif- 
fer’s proof in 1967 that the far-field pattern uo(%,d) for ,d ¢€ S? uniquely 
determines the shape of a sound-soft obstacle D. Unfortunately, Schiffer’s proof 
does not immediately generalize to other boundary conditions. This problem was 
remedied by Kirsch and Kress in 1993 who, using an idea originally proposed by 
Isakov, showed that uoo(%, d) for ,d € S* uniquely determines the shape of D 
as long as the solution of the direct scattering problem depends continuously on the 
boundary data [54]. In particular, it is not necessary to know the boundary condition 
a priori in order to guarantee uniqueness! The uniqueness problem for inverse 
scattering by an inhomogeneous medium was solved by Nachman [68], Novikov 
[70], and Ramm [79] in 1988 who based their analysis on the fundamental work of 
Sylvester and Uhlmann [88]. Their uniqueness proof was subsequently considerably 
simplified by Hahner [35]. 

The first attempt to reconstruct the shape of a sound-soft scattering obstacle 
from a knowledge of the far-field pattern in a manner acknowledging the nonlinear 
and ill-posed nature of the problem was made by Roger in 1981 using Newton’s 
iteration method [81]. A characterization and rigorous proof of the existence of 
the Fréchet derivative of the solution operator in Newton’s method were then 
established by Kirsch [47] and Potthast [75] in 1993 and 1994, respectively. An 
alternative approach to solving the inverse scattering problem was proposed by 
Colton and Monk in 1986 and by Kirsch and Kress in 1987 who broke up the inverse 
scattering problem into a linear, ill-posed problem and a nonlinear, well-posed 
problem [24, 53]. The optimization method of Kirsch and Kress has the attractive 
feature of needing only a single incident field for its implementation. On the other 
hand, to use such methods, it is necessary to know the number of components of the 
scatterer as well as the boundary condition satisfied by the field on the surface of 
the scatterer. These problems were overcome by Colton and Kirsch in 1996 through 
the derivation of a linear integral equation with the far-field data as its kernel (i.e., 
multistatic data is needed for its implementation) [19]. This method, subsequently 
called the linear sampling method, was further developed by Colton et al. [29] 


654 D. Colton and R. Kress 


and numerous other researchers. A significant development in this approach to 
the inverse scattering problem was the introduction of the factorization method by 
Kirsch in 1998 [48]. For further historical information on these “sampling” methods 
in inverse scattering theory, we refer the reader to the chapter in this handbook on 
sampling methods as well as the monographs [7,52]. 

Optimization methods and sampling methods for the inverse scattering problem 
for inhomogeneous media have been extensively investigated by numerous authors. 
In general, the optimization methods are based on rewriting the scattering problem 
corresponding to (9) as the Lippmann—Schwinger integral equation 


2 ik|x—y| 
ikxed _ k é 


u(x) =e m(y)u(y)dy, x €R, (11) 


4x Jn |x — yl 
where m := | —n and the object is to determine m from a knowledge of 


2 
Uoo (X, d) = Je. et "Ym(y)u(y) dy, %,d € S?. (12) 
An R3 
On the other hand, sampling methods have also been used to study the inverse 
scattering problem associated with (9) where now the object is to only determine 
the support of m. For details and further references, see [7,22,52]. 

Finally, as pointed out in [21], an alternative direction in inverse scattering theory 
than that discussed above is to only try to obtain lower and upper bounds on a 
few relevant features of the scattering object rather than attempting a complete 
reconstruction. This relatively new direction in inverse scattering theory will be 
discussed in Sect. 5. 


2 Direct Scattering Problems 
The Helmholtz Equation 


Most of the basic properties of solutions to the Helmholtz equation (3) can be 
deduced from the fundamental solution 


1 etkls-yl 


@(x, y) := 7 at xy. (13) 
a |x—y| 


For fixed y € R’, it satisfies the Helmholtz equation in R*\{y}. In addition, it 
satisfies the radiation condition (5) uniformly with respect to y on compact subsets 
of IR*. Physically speaking, the fundamental solution represents an acoustic point 
source located at the point y. In addition to plane waves, point sources will also 
occur as incident fields in scattering problems. 

Green’s integral theorems provide basic tools for investigating the Helmholtz 
equation. As an immediate consequence, they imply the Helmholtz representation 
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du dP(x, 
us) = fy 0) 2.9) — uo) GO” 


avg) ds(y), xeE D, (14) 


for solutions u € C?(D) N C'(D) to the Helmholtz equation. The representation 
(2) implies that solutions to the Helmholtz equation inherit analyticity from the 
fundamental solution. Any solution u to the Helmholtz equation in D satisfying 


u= —=0 on? (15) 


for some open subset T C dD must vanish identically in D. This can be seen via 
extending the definition of u by (2) for x € (IR?\D) UT. Then, by Green’s integral 
theorem, applied to u and ®(x, -), we have u = 0 in R?\D. Clearly u solves the 
Helmholtz equation in (IR*\D) UT and therefore by analyticity u = 0 in D since 
D and R3\D are connected through the gap I in 0D. 

As a consequence of the radiation condition (5), the Helmholtz representation is 
also valid in the exterior domain R? \D, i.e., We have 


0 0 = 
ux) =f fe u(y) PY) (2, »} ds(y), x€R\D, (16) 
dv(y) 


for radiating solutions u € C?(IR?\D) N C!(R3\D) to the Helmholtz equation. 
From Eq. (4) we observe that radiating solutions u to the Helmholtz equation satisfy 
Sommerfeld’s finiteness condition 


u(x) = o(a ), |x| + oo, (17) 


uniformly for all directions and that the validity of the Sommerfeld radiation 
condition (5) is invariant under translations of the origin. 

We are now in a position to introduce the fundamental notion of the far-field 
pattern of radiating solutions to the Helmholtz equation. 


Theorem 1. Every radiating solution u to the Helmholtz equation has an asymp- 
totic behavior of the form 


eiklx x| 1 
ula) = Jul) + 0 (= i Pee (18) 


uniformly in all directions X = x/|x|, where the function Ugo defined on the unit 
sphere S* is called the far-field pattern of u. Under the assumptions of (16), we 
have 


. 1 dewikiry gg _ipdey is 
Ho) = of | u(y) ———— a0) Honest ay &eES*. (19) 
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Proof. This follows from (16) by using the estimates 


eiklx—y| efklxl ees 27 ( 1 )| a eiklx—yl 
———__ = —— le @ : 
Ix—y| |x| |x| dv(y) |x— yl 
ik|x| a —ikxX+y 1 
e e 
ero 
me | mo)” ° in | 


which hold uniformly for all y € 0D and all directions x/|x| as |x| > oo. a 


From the representation (19), it follows that the far-field pattern is an analytic 
function on S?. As an extension of (18), each radiating solution u to the Helmholtz 
equation has an Atkinson—Wilcox expansion of the form 


et k lx! oo 
u(x) = ae ; r(2 ) (20) 


that converges absolutely and uniformly on compact subsets of R3\ B, where 
B > D is a ball centered at the origin. The coefficients in the expansion (20) are 
determined in terms of the far-field pattern Fy = ugg by the recursion 


QklFy = (0-1) Fey + BFr1, €=1,2,..., (21) 


where B denotes the Laplace—Beltrami operator for the unit sphere. The following 
consequence of the expansion (20) is known as Rellich’s lemma. 


Lemma 1. Let u be a radiating solution to the Helmholtz equation for which the 
far-field pattern Ugg vanishes identically. Then u vanishes identically. 


Proof. This follows from (20) and (21) together with the analyticity of solutions to 
the Helmholtz equation. a 


Corollary 1. Let u € C2(R*\D) M C!(R3\D) be a radiating solution to the 
Helmholtz equation in R3\D for which 


im f ut ds 20. (22) 


Then u = 0 in R°\D. 


Proof. Using Green’s integral theorem, the radiation condition can be utilized to 
establish that 
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du |? dit 
tim, f ) 4 lus ds = —2kIm | ou as. 
TOS yay av dD dv 
Now the assumption (22) implies lim,—,o9 Jix = |u\>ds = 0, and the statement 
follows from Lemma 1. wi 


Scattering from infinitely long cylindrical obstacles or inhomogeneities leads 
to the Helmholtz equation in R?. The two-dimensional case can be used as an 
approximation for scattering from finitely long cylinders, and more importantly, 
it can serve as a model case for testing numerical approximation schemes in direct 
and inverse scattering. Without giving details, we can summarize that our analysis 
remains valid in two dimensions after appropriate modifications of the fundamental 
solution and the radiation condition. The fundamental solution to the Helmholtz 
equation in two dimensions is given by 


I 
P(x, y) = 7 AS (klx—yl), x Fy, (23) 
dd) 


in terms of the Hankel function Hp’ of the first kind of order zero. In R? the 
Sommerfeld radiation condition has to be replaced by 


lim vi (5 2) = eee. (24) 
roo r 


uniformly for all directions x/|x|. According to the form (24) of the radiation 
condition, the definition of the far-field pattern (18) has to be replaced by 


eiklx| 
u(x) = Ta Jo +0 (— .)} , |x| > o, (25) 


and the representation (19) assumes the form 


elf eT ik x: y 


Ugo (X) = Waa 


ou 


| re cial v0) Ay 


(y) me ds(y) (26) 
for X = x/|x|. 


Obstacle Scattering 


After renaming the unknown functions, the direct scattering problem for sound- 
soft obstacles is a special case of the following exterior Dirichlet problem: Given a 
function f € C(dD), find a radiating solution u € C?(IR°\D) NM C(IR3\D) to the 
Helmholtz equation that satisfies the boundary condition 
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u= f ondD. (27) 


Theorem 2. The exterior Dirichlet problem for the Helmholtz equation has at most 
one solution. 


Proof. Let u satisfy the homogeneous boundary condition u = 0 on OD. If u 
were continuously differentiable up to the boundary, we could immediately apply 
Corollary 1 to obtain u = 0 in R°>\D. However, in the formulation of the exterior 
Dirichlet problem, u is only assumed to be in C(IR°\D). We refrain from discussing 
possibilities to overcome this regularity gap and refer to the literature [22]. B 


Theorem 3. The exterior Dirichlet problem has a unique solution. 
Proof. The existence of a solution can be elegantly based on boundary integral 


equations. In the layer approach, one tries to find the solution in the form of a 
combined acoustic double- and single-layer potential 


_ dD(x,y) 
FO [Se -ie6.9 oly) ds(y) (28) 


for x € R3\D with a density g € C(dD). Then, after introducing the single- and 
double-layer integral operators S, K : C(0@D) > C(aD) by 


(Sp)(x) =2 [, H(x, vol) ds(y), x € dD, (29) 
OP(x, 
(Ko)(x) = 2 [SEH oor as0. x€ 0D, (30) 


and using their regularity and jump relations, it can be seen that (28) solves the 
exterior Dirichlet problem provided the density g is a solution of the integral 
equation 


g+ Kg—iSg =2f. (31) 


Due to their weakly singular kernels, the operators S and K turn out to be compact. 
Hence, the existence of a solution to (31) can be established with the aid of the 
Riesz—Fredholm theory for compact operators by showing that the homogeneous 
form of (31) only allows the trivial solution g = 0. 

Let ¢g be a solution of the homogeneous equation, and let the subscripts + denote 
the limits obtained by approaching 0D from R3\D and D, respectively. Then the 
potential uw defined by (28) in all of IR*\dD satisfies the homogeneous boundary 
condition wu; = 0 on dD whence u = 0 in R*\D followed by Theorem 2. The 
jump relations for single- and double-layer potentials now yield 
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Hence, using Green’s first integral theorem, we obtain 


if joPds = [ i as = | {| grad u|* — k |u|? dx, 
aD aD dv D 


and taking the imaginary part yields g = 0. a 


We note that, in addition to existence of a solution, the Riesz—Fredholm theory 
also establishes well-posedness, i.e., the continuous dependence of the solution on 
the data. Instead of the classical setting of continuous function spaces, the existence 
analysis can also be considered in the Sobolev space H'/?(D) for the boundary 
integral operators leading to solutions in the energy space He. (R3\D) (see [64, 
69]). 

We further note that without the single-layer potential included in (28), the 
corresponding double-layer integral equation suffers from nonuniqueness if k 
is a so-called irregular wave number or internal resonance, i.e., if there exist 
nontrivial solutions u to the Helmholtz equation in the interior domain D satisfying 
homogeneous Neumann boundary conditions du/dv = 0 on oD. 

For the numerical solution of the boundary integral equations in scattering theory 
via spectral methods in two and three dimensions, we refer to [22]. For boundary 
element methods, we refer to [80]. 

In general, for the scattering problem, the boundary values are as smooth as 
the boundary, since they are given by the restriction of the analytic function u' to 
dD. Therefore, we may use the Helmholtz representation (16) and Green’s second 
integral theorem applied to uv’ and ®(x, -) to obtain the following theorem. 


Theorem 4. For scattering from a sound-soft obstacle D, we have 


u(x) =v) — / Oe. yds), ERD, (32) 


and the far-field pattern of the scattered field u’ is given by 


u 


tool) = i BH yy oF As(y), BES? (33) 
Iv aD Ov 


The representation (32) for the scattered field through the so-called secondary 
sources on the boundary is known as Huygens’ principle. Here we will use it for 
the motivation of the Kirchhoff or physical optics approximation as an intuitive 
procedure to simplify the direct scattering problem. For large wave numbers &, i.e., 
for small wave lengths, in a first approximation, a convex object D locally may be 
considered at each point x of dD as a plane with normal v(x). This suggests 


660 D. Colton and R. Kress 


du ou! 
oe). 
dv ov 


on the part DD_ := {x € 0D: v(x)-d < 0} that is illuminated and 


du r 
dv 
in the shadow region dD+ := {x € 0D : v(x)-d > O}. Thus, the Kirchhoff 


approximation for the scattering of a plane wave with incident direction d at a 
convex sound-soft obstacle is given by 


deik yd 


ux) wet? — 2f Hey (x, y) ds(y) (34) 


for x € R>\D. 


Scattering by an Inhomogeneous Medium 


Recall the scattering problem for an inhomogeneous medium with refractive index 
n_ as described by (9) for the total wave u = u' + u° with incident field u' and 
the scattered wave u* satisfying the Sommerfeld radiation condition. The function 
m := 1—mnhas support D. 

The counterpart of the Helmholtz representation is given by the Lippmann— 
Schwinger equation 


ux) =O) —# i. B(x, yym(y)u(y) dy, x € BY, (35) 


which can be shown to be equivalent to the scattering problem. 

In order to establish existence of a solution to (35) via the Riesz—Fredholm 
theory, it must be shown that the homogeneous equation has only the trivial solution 
or, equivalently, that the only solution to (9) satisfying the radiation condition is 
identically zero. For this, in addition to Rellich’s lemma, the following unique 
continuation principle is required: Any solution u € C?(G) of Eq. (9) in a domain 
G C R’ such that n € C(G) and u vanishes in an open subset of G vanishes 
identically. Hence, we have the following result on existence and uniqueness for the 
inhomogeneous medium scattering problem. 


Theorem 5. For a refractive index n € C'(IR*) with Ren > 0 and Imn > 0, 
the Lippmann—Schwinger equation or, equivalently, the inhomogeneous medium 


scattering problem has a unique solution. 


Proof. From Green’s first integral theorem, it follows that 
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- 
i i as= | {| grad u|? — k2A|u|?} dx. 
aD i) D 


Vv 


Taking the imaginary part and applying Corollary | yields u = 0 in R°\D in view of 
the assumptions on 1, and the proof is finished by the unique continuation principle. 
H 


From (35) we see that 
w(x) = f(x, you) dy, x ER’. 
R3 


Hence, the far-field pattern ugg is given by 


ek* Ym(y)u(y)dy, & € S?. (36) 


x) = —— 
Hoo ) An R3 
We note that for k*||77||oo sufficiently small, u can be obtained by the method of 


successive approximations. If in (36) we replace u by the first term in this iterative 
process, we obtain the Born approximation 


i ke EASY i 
Ugo (&) -= fe kX Ym(y)ui(y)dy, % € S?. (37) 


For numerical solutions of the inverse medium scattering problem by finite 
element methods coupled with boundary element methods via nonlocal boundary 
conditions, we refer to [65]. 


The Maxwell Equations 


We now consider the Maxwell equations as the foundation of electromagnetic 
scattering theory. Our presentation is organized parallel to that on the Helmholtz 
equation, i.e., on acoustic scattering, and will be confined to homogeneous isotropic 
media. Consider electromagnetic wave propagation in an isotropic dielectric 
medium in R? with constant electric permittivity « and magnetic permeability 
jt. The electromagnetic wave is described by the electric field € and the magnetic 
field H satisfying the time-dependent Maxwell equations 


ole +4 2 = 0, cule = =0. (38) 


For time-harmonic electromagnetic waves of the form 


a,j) =Rete CEGje 4. Fie) =Rele awe} (39) 
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with frequency w > 0, the complex-valued space-dependent parts E and H satisfy 
the reduced Maxwell equations 


cullA—-—ikH =0, curlH +ikE =0, (40) 


where the wave number k is given by the positive constant k = ,/e@. We will 
only be concerned with the reduced Maxwell equations and will henceforth refer to 
them as the Maxwell equations. 

A solution EF’, H to the Maxwell equations whose domain of definition contains 
the exterior of some sphere is called radiating if it satisfies one of the Silver—Miiller 
radiation conditions 


lim (H x x —rE) =0 (41) 
ro 
or 
lim (E x x +rH) =0, (42) 
roo 


where r = |x| and the limits hold uniformly in all directions x /|x|. For more details 
on the physical background of electromagnetic waves, we refer to [46, 67]. 

For the Maxwell equations, the counterpart of the Helmholtz representation (2) 
is given by the Stratton—Chu formula 


E(x) =—corl iy | 20) x EQ) &(x, 9) d5(9) 


1 (43) 
+e curl curl f v(y) x A(y) ®(x, y)ds(y), x € D, 
I aD 


for solutions E, H € C!(D)M C(D) to the Maxwell equations. A correspond- 
ing representation for H can be obtained from (43) with the aid of H = curl E/ik. 

The representation (43) implies that each continuously differentiable solution to 
the Maxwell equations automatically has analytic Cartesian components. Therefore, 
one can employ the vector identity curlcurl E = —AE + grad div £ to prove that 
for a solution FE, H to the Maxwell equations, both & and H are divergence-free 
and satisfy the vector Helmholtz equation. Conversely, if E is a solution to the 
vector Helmholtz equation AE + k?E = 0 satisfying divE = 0, then E and 
H := curl E/ik satisfy the Maxwell equations. 

It can be shown that solutions E’, H to the Maxwell equations in D satisfying 


vxE=vxH=0 onl (44) 
for some open subset  C dD must vanish identically in D. 


As a consequence of the Silver—Miiller radiation condition, the Stratton—Chu 
formula is also valid in the exterior domain R3\D, i.e., we have 
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E(x) =curl / | 20) E() BC, 9) ds) 


(45) 
“| curl eur f v(y) x H(y) ®(x, y)ds(y), x € R°\D, 
aD 


for radiating solutions E, H € C'(IR*\D) M C(R3\D) to the Maxwell equations. 
Again, a corresponding representation for H can be obtained from (45) with the aid 
of H = curl E/ik. 

From (45) it can be seen that the radiation condition (41) implies (42) and vice 
versa. Furthermore, one can deduce that radiating solutions E, H to the Maxwell 
equations automatically satisfy the Silver—Miiller finiteness conditions 


E(x) =O (Z). H(x) =O (—). |x| > 00, (46) 
[>| [>| 


uniformly for all directions and that the validity of the Silver—Miiller radiation 
conditions (41) and (42) is invariant under translations of the origin. From the 
Helmholtz representation (16) for radiating solutions to the Helmholtz equation and 
the Stratton—Chu formulas for radiating solutions to the Maxwell equations, it can 
be deduced that for solutions to the Maxwell equations, the Silver—Miiller radiation 
condition is equivalent to the Sommerfeld radiation condition for the Cartesian 
components of FE and H. 

The Stratton—Chu formula (45) can be used to introduce the notion of the electric 
and magnetic far-field patterns. 


Theorem 6. Every radiating solution E,H to the Maxwell equations has the 
asymptotic form 


ik|x| 


ik|x| 
E(x) = ae | Boo) +0 (=) . A(x)= a | Hast) +0 (=) 
|x| || || |x| 
(47) 


for |x| — oo uniformly in all directions X = x/\x|, where the vector fields Egg 
and Ho defined on the unit sphere S* are called the electric far-field pattern and 
magnetic far-field pattern, respectively. They satisfy 


Ho =VX Eq and v-E~ =v: He =0 (48) 


with the unit outward normal v on S*. Under the assumptions of (45), we have 


An ik An An —ik Rey 
Bool) = 8x f (00) x EQ) + BO) x HON] x} HF ds) 
D 
(49) 
for & € S* and a corresponding expression for Hoo. 


Rellich’s lemma carries over immediately from the Helmholtz to the Maxwell 
equations. 
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Lemma 2. Let E, H be a radiating solution to the Maxwell equations for which the 
electric far-field pattern Ego vanishes identically. Then E and H vanish identically. 


The electromagnetic counterpart of Corollary | is given by the following result. 


Corollary 2. Let E,H € C!(IR°\D) 9 C(R*\D) be a radiating solution to the 
Maxwell equations in R3>\D for which 


Re | v-Ex Hds <0. 
aD 


Then E = H =O inR?\D. 
For two vectors d, p € R? with |d| = 1 and p-d = 0, the plane waves 

E'(x) = peke'@, Hi (x) =dx peik*'4 (50) 
satisfy the Maxwell equations for all x € IR*. The orthogonal vectors p and 
d x p describe the polarization direction of the electric and the magnetic field, 
respectively. Given the incident field E’, H' and a bounded domain D C R’, 
the simplest obstacle scattering problem is to find the scattered field E*, H* as a 
radiating solution to the Maxwell equations in the exterior of the scatterer D such 


that the total field E = E' + E°, H = H' + H* satisfies the perfect conductor 
boundary condition 


vxE=0 ondoD, (51) 


where v is the outward unit normal to dD. A more general boundary condition is 
the impedance or Leontovich boundary condition 


vx H-A(vx E)xv=0 onda, (52) 
where A is a positive constant or function called the surface impedance. 
Theorem 7. The scattering problem for a perfect conductor has a unique solution. 
Proof. Uniqueness follows from Corollary 2. The existence of a solution again can 


be based on boundary integral equations. In the layer approach, one tries to find the 
solution in the form of a combined magnetic and electric dipole distribution 


E%(x) = curl f a(y) ®(x, y) ds(y) 


(53) 
+i curleurt v(y) x (Spa) (*)®, y) ds(y), x € R8\aD. 
aD 
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Here So denotes the single-layer operator (29) in the potential theoretic limit 
case k = 0, and the density a is assumed to belong to the space Ci“ (aD) of 
Holder continuous tangential fields with Hélder continuous surface divergence. 
After defining the electromagnetic boundary integral operators M and N by 


(Ma)(x) := 2 | v(x) x curl, {a(y) P(x, y)} ds(y), x € aD, (54) 
aD 
and 
(Na)(x) := 2 v(x) x curleur f v(y) x a(y) ®(x, y)ds(y), x EdD, (55) 
aD 


it can be shown that E* given by (53) together with H* = curl E*/ik solves the 
perfect conductor scattering problem provided the density a satisfies the integral 
equation 


a+Ma+iNPSja=—2vx E'. (56) 


Here the operator P is defined by Pb := (v xb) x v for (not necessarily tangential) 
vector fields b. Exploiting the smoothing properties of the operator So, it can be 
shown that M + iN PS? is a compact operator from ci (aD) into itself. The 
existence of a solution to (56) can now be based on the Riesz—Fredholm theory 
by establishing that the homogeneous form of (56) only has the trivial solution [22]. 
H 


Note that, analogous to the acoustic case, without the electric dipole distribution 
included in (53), the corresponding magnetic dipole integral equation is not uniquely 
solvable if & is a irregular wave number, i.e., if there exists a nontrivial solution 
E,H to the Maxwell equations in D satisfying the homogeneous boundary 
condition v x E = OondD. 

Instead of the classical setting of Hélder continuous functions, the integral 
equation can also be considered in the Sobolev space H. 1? (@D) of tangential fields 
in H'/2(dD) that have a weak surface divergence in H'/*(dD) (see [69]). 

In addition to electromagnetic obstacle scattering, one can also consider scat- 
tering from an inhomogeneous medium where outside a bounded domain D the 
electric permittivity e and magnetic permeability jz are constant and the conductivity 
o vanishes, i.e., ¢ = £0, 6 = fo ando = 0 in R3\D. For simplicity we will assume 
that the magnetic permeability is constant throughout R?. Then, again assuming the 
time-harmonic form (39) with ¢ and yz replaced by &9 and j10, respectively, the total 
fields E = E'+ E°, H = H' + Hi‘ satisfy 


culE —ikH =0, culH +iknE=0 inR? (57) 
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and the scattered wave EL’, H® satisfies the Silver—Miiller radiation condition, where 
the wave number is given by k = ,/egHow andn = (¢€+i0/w@) /é is the 
refractive index. Establishing uniqueness requires an electromagnetic analogue of 
the unique continuation principle, and existence can be based on an electromagnetic 
variant of the Lippmann—Schwinger equation [22]. 

The scattering of time-harmonic electromagnetic waves by an infinitely long 
cylinder with a simply connected cross section D leads to boundary value problems 
for the two-dimensional Helmholtz equation in the exterior R°\D of D. If the 
electric field is polarized parallel to the axis of the cylinder and if the axis of the 
cylinder is parallel to the x3 axis, then 


1 du ou 
E=(0,0,4), H=— 7 
m0,0sa) ik (= dx 0) 


satisfies the Maxwell equations if and only if uv = u(x,, x2) satisfies the Helmholtz 
equation. The homogeneous perfect conductor boundary condition is satisfied on the 
boundary of the cylinder if the homogeneous Dirichlet boundary condition vu = 0 on 
dD is satisfied. If the magnetic field is polarized parallel to the axis of the cylinder, 
then the roles of EF and H have to be reversed, i.e., 


i { ou ou 
H=(0,0,0), E= -—.0), 
ie =(= Ox, 0) 


and the perfect conductor boundary condition corresponds to the Neumann bound- 
ary condition du/dv = 0 on dD with the unit normal v to the boundary dD of the 
cross section D. Hence, the analysis of two-dimensional electromagnetic scattering 
problems coincides with that of two-dimensional acoustic scattering problems. 


Historical Remarks 


Equation (4) carries the name of Helmholtz (1821-1894) for his contributions 
to mathematical acoustics. The radiation condition (5) was introduced by Som- 
merfeld in 1912 to characterize an outward energy flux. The expansion (20) was 
first established by Atkinson in 1949 and generalized by Wilcox in 1956. The 
fundamental Lemma | is due to Rellich (1943) and Vekua (1943). The combined 
single- and double-layer approach (28) for the existence analysis was introduced 
independently by Leis, Brakhage and Werner, and Panich in the 1960s in order to 
remedy the nonuniqueness deficiency of the classical double-layer approach due to 
Vekua, Weyl, and Miiller from the 1950s. Huygens’ principle is also referred to as 
the Huygens—Fresnel principle and named for Huygens (1629-1695) and Fresnel 
(1788-1827) in recognition of their contributions to wave optics. The physical 
optics approximation (34) is named for Kirchhoff (1824-1887) for his contributions 
to optics. The terms Lippmann—Schwinger equation and Born approximation are 
adopted from quantum physics. Equation (38) is named for Maxwell (1831- 
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1879) for his fundamental contributions to electromagnetic theory. The radiation 
conditions (41) and (42) were independently introduced in the 1940s by Silver and 
Miiller. The integral representations (43) and (45) were first presented by Stratton 
and Chu in 1939. Extending the ideas of Leis, Brakhage and Werner, and Panich 
from acoustics to electromagnetics, the approach (53) was introduced independently 
by Knauff and Kress, Jones, and Mautz and Harrington in the 1970s in order to 
remedy the nonuniqueness deficiency of the classical approach due to Weyl and 
Miiller from the 1950s. 


3 Uniqueness in Inverse Scattering 
Scattering by an Obstacle 


The first step in studying any inverse scattering problem is to establish a uniqueness 
result, i.e., if a given set of data is known exactly, does this data uniquely determine 
the support and/or the material properties of the scatterer? We will begin with the 
case of scattering by an impenetrable obstacle and then proceed to the case of a 
penetrable obstacle. 

From section “Obstacle Scattering,” we recall that the direct obstacle scattering 
problem is to find u € C?(IR°\D) M C(IR3\ D) such that the total field u = u! + uw’ 
satisfies the Helmholtz equation 


Au+tk?u=0 inR?\D (58) 
and the sound-soft boundary condition 
u=0O ondD, (59) 


where u! (x) = e'***4, |d| = 1, and w’ is a radiating solution. We also recall from 
Theorem | that u* has the asymptotic behavior 


ik|x| 1 
e a 
u’(x,d) = <- | std.) +0(——)t , |x| > 0, (60) 
|x| |x| 
uniformly for all directions x = x/|x|, where Ugo is the far-field pattern of the 


scattered field E°. By Green’s integral theorem and the far-field representation (33), 
it can be shown that the far-field pattern satisfies the reciprocity relation [22] 


ting (%, 2) = teg(=a,=2),. Xd € 8. (61) 


The inverse scattering problem we are concerned with is to determine D froma 
knowledge of woo (X, d) for & and d on the unit sphere S* and fixed wave number 
k. In particular, for the acoustic scattering problem (58) and (59), we want to show 
that D is uniquely determined by uoo(X,d) for &,d € S*. We note that by the 
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reciprocity relation (61), the far-field pattern uo is an analytic function of both * 
and d, and hence it would suffice to consider uo for X and d restricted to a surface 
patch of the unit sphere S?. 


Theorem 8. Assume that D, and D> are two obstacles such that the far-field 
patterns corresponding to the exterior Dirichlet problem (58) and (59) for D, and 
Dy coincide for all incident directions d. Then D, = Do. 


Proof. Let uj and wu be the scattered fields corresponding to D; and Dz, respec- 
tively. By the analyticity of the scattered field as a function of x and Rellich’s 
Lemma 1, the scattered fields satisfy uj(.,d) = u5(-,d) in the unbounded 
component G of the complement of D; U D3 for all d € S”. This in turn implies 
that the scattered fields corresponding to ®(-, z) as incident field and D, or D2 as 
the scattering obstacle satisfy u}(x,z) = u3(x,z) for all x,z € G. Now assume 
that D; #~ Dp». Then, without loss of generality, there exists x* € 0G such 


that x* € 0D, and x* ¢ Dy. Then setting z, := x* + Lu(x*), we have that 
limy—+o0 U5 (X*, Zn) exists but limp 99 uy (X*, Z1) = 00 which is a contradiction and 
hence D,; = Do. |_| 


An open problem is to determine if one incident plane wave at a fixed wave 
number k is sufficient to uniquely determine the scatterer D. If it is known a priori 
that in addition to the sound-soft boundary condition (59) D is contained in a ball of 
radius R such that kR < 4.49, then D is uniquely determined by its far-field pattern 
for a single incident direction d and fixed wave number k [32] (see also [22]). D 
is also uniquely determined if instead of assuming that D is contained in a ball of 
sufficiently small radius, it is assumed that D is close to a given obstacle [87]. It is 
also known that for a wide class of sound-soft scatterers, a finite number of incident 
fields are sufficient to uniquely determine D [82]. Finally, if it is assumed that D 
is polyhedral, then a single incident plane wave is sufficient to uniquely determine 
D [1,63]. 

We conclude this section on uniqueness results for the inverse scattering problem 
for an obstacle by considering the scattering of electromagnetic waves by a perfectly 
conducting obstacle D. From section “The Maxwell Equations” we recall that the 
direct obstacle scattering problem is to find E, H € C?(IR*\D) MN C(R3\D) such 
that the total field E = E' + E°, H = H' + H’ satisfies the Maxwell equations 


culE—-ikH =0, culH+ikE=0 inR*\D (62) 
and the perfect conductor boundary condition 
vx E=0 ona, (63) 


where E', H' is the plane wave given by (50) and E°, H° is a radiating solution. 
We also recall from Theorem | that u* has the asymptotic behavior 
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eb kx 


|x| 


E*(x,d,p) = Eoo(X,d,p) +O (—) , |x| >, (64) 


|x| 


where Fg, is the electric far-field pattern of the scattered field E°. 

The inverse scattering problem is to determine D from a knowledge of 
Eo.(&,d,p) for & and d on the unit sphere $7, three linearly independent 
polarizations p, and fixed wave number k. We note that Fo is an analytic function 
of x and d and is linear with respect to p. The following theorem can be proved 
using the same ideas as in the proof of Theorem 8. 


Theorem 9. Assume that D, and D2 are two perfect conductors such that for a 
fixed wave number k, the electric far-field patterns for both scatterers coincide for 
all incident directions d and three linearly independent polarizations p. Then D, = 
Dy. 


In the case when D consists of finitely many polyhedra, a single incident wave 
is sufficient to uniquely determine D [62]. 


Scattering by an Inhomogeneous Medium 


We now return to scattering of acoustic waves, but instead of scattering by a sound- 
soft obstacle, we consider scattering by an inhomogeneous medium where the 
governing equation (see section “Scattering by an Inhomogeneous Medium’) is 


Au+k?nu=0 in R? (65) 


for u = u' + wu’ € C?(IR?), where n € C'!(IR°) is the refractive index satisfying 
Ren > OandImn > 0, w(x) = e'***4 and wu! is radiating. We let D denote 
the support of m := 1 —n. By Theorem | the scattered wave u° again has the 
asymptotic behavior (60). The inverse scattering problem we are now concerned 
with is to determine the index of refraction n (and hence D) from a knowledge of 
Ugo(X, d) for ¥ and d on the unit sphere S* and fixed wave number k. In particular, 
we want to show that n is uniquely determined from ug9(X,d) for x,d € S 2 and 
fixed wave number k. 


Theorem 10. The refractive index n in (65) is uniquely determined by uco(X, a) for 
&,d € S* and a fixed value of the wave number k. 


Proof. Let B be an open ball centered at the origin and containing the support of 


m = | —n. The first step in the proof is to construct a solution of (65) in B of the 
form 


w(x) =e *(1+ r(x), (66) 
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where z-z=0, ze C?, and 


C 
r|lp203) < —— 
Ir lize) [Rez] 


for some positive constant C and | Re z| sufficiently large. This is done in [35] by 
using Fourier series. The second step is to show that, given two open balls B; and 
B> centered at the origin and containing the support of m such that B; C Bz, the set 
of solutions {u( d):ideé S?\ satisfying (65) is complete in 


H := {we C*(B)): Aw+k?nw =0_ in By} 


with respect to the norm in L*(B;) [35]. Now assume that n; and n2 are 
refractive indices such that the corresponding far-field patterns satisfy U10(-,d) = 
U2,00(-,d),d € S*, and assume that the supports of 1 — n; and 1 — nz are contained 
in B,. Then using Rellich’s Lemma | and Green’s integral theorem, it can be shown 
that 


[0 Dua.dyin 19) dx = 0 
By 
for all d, d € S? and hence 
/ w1W2(ny — N2)dx =0 (67) 
By 


for all solutions w;,w2 € C?(B2) of Aw; + k2nyw = Oand Aw. + k?now2 = 0 
in By. Now choose z; := y + pa + ib and z := y — pa —ib such that {y,a,b} 
is an orthogonal basis in IR? with the properties that |a| = 1 and |b|? = |y|? + p?, 
and substitute these values of z into (66) arriving at functions w; and w2. Substitute 
these functions into (67), and let p — oo to arrive at 


i e%¥°* (ni (x) —n2(x)) dx =0 
By 


for arbitrary y € R?, i.e.,n4(x) = n2(x) for x € By by the Fourier integral theorem. 
a 


In the case of scattering by a sound-soft obstacle, the proof of uniqueness given 
in Theorem 8 remains valid in R?. However, this is not the case for scattering 
by an inhomogeneous medium. Indeed, until recently, the question of whether or 
not Theorem 10 remains valid in R? was one of the outstanding open problems in 
inverse scattering theory. The problem was finally resolved in 2008 by Bukhgeim [4] 
who showed that in R? the index of refraction n is uniquely determined by uo (%, d) 
for X,d € S' anda fixed value of the wave number k. 
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We conclude this section with a few remarks on scattering by an anisotropic 
medium. Let n be as above and recall that D is the support of m := 1 —n. Let A 
be a 3 x 3 matrix-valued function whose entries a ;, are continuously differentiable 
functions in D such that A is symmetric and satisfies 


E-(ImA)E <0, &-(Re A)E > y|é/? 


for allé € C and x € D, where y is a positive constant. We assume that A(x) = J 
for x € R3\D. The anisotropic scattering problem is to find u = u! +u € Hh R? ) 
such that 


V-AVu+k?nu=0 in R? (68) 
in the weak sense where again u' (x) = e’***4 and uw’ is radiating. The existence of 
a unique solution to this scattering problem has been established by Hahner [36]. 

The scattered field again has the asymptotics (60). The inverse scattering problem 
is now to determine D froma knowledge of the far-field pattern ug9(X, d) for %,d € 
S?. We note that the matrix A is not uniquely determined by uo, and hence without 
further a priori assumptions, the determination of D is the most that can be hoped 
for [33,74]. To this end we have the following theorem due to Hahner [36]. 


Theorem 11. Assume y > 1. Then D is uniquely determined by ugg(X,d) for 
xd € S?, 


We note that Theorem 11 remains valid if the condition on Re A is replaced by 
the condition 


&-(ReA JE > plé|’ 


for all € € C? and x € D where yp is a positive constant such that jz > 1 [7]. Note 
that the isotropic case when A = J is handled by Theorem 10. 

Uniqueness theorems for the Maxwell equations in an isotropic inhomogeneous 
medium have been established by Colton and Paivarinta [27] and Hahner [37]. The 
proof is similar to that of Theorem 10 for the scalar problem except that technical 
problems arise due to the fact that we must now construct a solution E', H to the 
Maxwell equations in an inhomogeneous isotropic medium such that E has the 
form 


E(x) =e™*(n+r(x)), 


where z, 7 € C3, n-z=0,andz-z= k?. In contrast to the case of acoustic waves, 
it is no longer true that r(x) decays to zero as | Re z| tends to infinity. Finally, the 
generalization of Theorem 11 to the case of the Maxwell equations in an anisotropic 
media has been done by Cakoni and Colton [5]. 
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Historical Remarks 


As previously mentioned, the first uniqueness theorem for the acoustic inverse 
obstacle problem was given by Schiffer in 1967 for the case of a sound-soft obstacle 
[61], whereas in 1988 Nachman [68], Novikov [70], and Ramm [79] established a 
uniqueness result for the inverse scattering problem for an inhomogeneous medium. 
In 1990 Isakov [40,41] proved a series of uniqueness theorems for the transmission 
problem with discontinuities of u across 0D. His ideas were subsequently utilized 
by Kirsch, Kress, and their coworkers to establish uniqueness theorems for a variety 
of inverse scattering problems for both acoustic and electromagnetic waves (for 
references, see [22]). In particular, the proofs of Theorems 8 and 9 are based on 
the ideas of Kirsch and Kress [22,54]. 

A global uniqueness theorem for the Maxwell equations in an isotropic inhomo- 
geneous medium was first established in 1992 by Colton and Paivarinta [27] (see 
also [37]). The results of [27,37] are for the case when the magnetic permeability 
4 is constant. For uniqueness results in the case when jz is no longer constant, we 
refer to [71,72]. 


4 Iterative and Decomposition Methods in Inverse 
Scattering 


Newton Iterations in Inverse Obstacle Scattering 


We now turn to reconstruction methods for the inverse scattering problem for sound- 
soft scatterers, and as a first group we describe iterative methods. Here the inverse 
problem is interpreted as a nonlinear ill-posed operator equation which is solved 
by iteration methods such as regularized Newton methods, Landweber iterations, 
or conjugate gradient methods. For a fixed incident field uv’, the solution to the 
direct scattering problem defines the boundary to far field operator F :0D +> Ugo 
which maps the boundary dD of the scatterer D onto the far-field pattern Ugo of the 
scattered wave u°. In particular, F is the imaging operator that takes the scattering 
object D into its image Ugo via the scattering process. In terms of this imaging 
operator, i.e., the boundary to far field operator, given a far-field pattern Ugo, the 
inverse problem consists in solving the operator equation 


F(8D) = te (69) 


for the unknown boundary dD. As opposed to the direct obstacle scattering problem 
which is linear and well-posed, the operator equation (69), i.e., the inverse obstacle 
scattering problem, is nonlinear and ill-posed. It is nonlinear since the solution to 
the direct scattering problem depends nonlinearly on the boundary, and it is ill-posed 
because the far-field mapping is extremely smoothing due to the analyticity of the 
far-field pattern. 
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In order to define the operator ¥ properly, the most appropriate approach is to 
choose a fixed reference domain D and consider a family of scatterers Dy, with 
boundaries represented in the form 0D), = {x++h(x):x € 0D}, where h: dD — R? 
is of class C? and is sufficiently small in the C* norm on 0D. Then we may consider 
the operator F as a mapping from a ball 


V := {he COD): ||hllc2 <a} C C*(aD) 


with sufficiently small radius a > 0 into L?(S*). However, for ease of presentation, 
we proceed differently and restrict ourselves to boundaries dD that can be parame- 
terized by mapping them globally onto the unit sphere S?, i-e., 


aD = { p(k): 8 € S*} (70) 


for some injective C? function p : S? — R?. As a simple example, the reader 
should consider the case of starlike domains where 


pPR)=r@s, Fe S?, (71) 


with a radial distance function r:S* —> (0,00). Then, with some appropriate 
subspace W Cc C?(S*), we may interpret the operator F as a mapping 


F:W—-L?(S*), Fp too, 
and consequently the inverse obstacle scattering problem consists in solving 
F(P) = Uco (72) 


for the unknown function p. 
Since ¥ is nonlinear we may linearize 


F(p +4) =F(p) + F(p)q + 0 (q’) 


in terms of a Fréchet derivative ¥’. Then, given an approximation p for the solution 
of (72), in order to obtain an update p + q, we solve the approximate linear equation 


F(p) + F'(p)q = too (73) 


for g. We note that the linearized equation inherits the ill-posedness of the nonlinear 
equation and therefore regularization is required. As in the classical Newton 
iterations, this linearization procedure is iterated until some stopping criteria is 
satisfied. 

In principle the parameterization of the update 0D ,+4 = {p(X)+q(&) : & € S7} 
is not unique. To cope with this ambiguity, the simplest possibility is to allow only 
perturbations of the form 
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q(&) = 2%) v(p(&)), x € S’, (74) 


with a scalar function z. We denote the corresponding linear space of normal L” 
vector fields by | aa fl *), 

The Fréchet differentiability of the operator F is addressed in the following 
theorem. 


Theorem 12. The boundary to far-field mapping F : p +> Ugo is Fréchet differen- 
tiable. The derivative is given by 


F'(p)q = Vq,00: 


where Vq,o0 is the far-field pattern of the radiating solution vg to Helmholtz equation 
in R3\D satisfying the Dirichlet boundary condition 


ou 
Vg =—-v-g ap on 0D (75) 


in terms of the total field u = ul + wu’. 


The boundary condition (75) for the derivative can be obtained formally by using 
the chain rule to differentiate the boundary condition u = 0 on dD with respect to 
the boundary. Extensions of Theorem 12 to the Neumann boundary condition, the 
perfect conductor boundary condition, and to the impedance boundary condition in 
acoustics and electromagnetics are also available. 

To justify the application of regularization methods for stabilizing (73), injec- 
tivity and dense range of the operator F’(p) : L?,.,(S7) — L?(S”) need to be 
established. This is settled for the Dirichlet condition and, for A sufficiently large, for 
the impedance boundary condition and remains an open problem for the Neumann 
boundary condition. In the classical Tikhonov regularization, (73) is replaced by 


aq +[F'(p)]° F'(p)a = [F'(p)]" {uco — F(p)} (76) 


with some positive regularization parameter a and the L? adjoint [F’(p)]* of F’(p). 
For details on the numerical implementation, we refer to [22] and the references 
therein. The numerical examples strongly indicate that it is advantageous to use 
some Sobolev norm instead of the L* norm as the penalty term in the Tikhonov 
regularization. Numerical examples in three dimensions have been reported by 
Farhat et al. [31] and by Harbrecht and Hohage [38]. 

In closing this section on Newton iterations, we note as their main advantages 
that this approach is conceptually simple and, as the numerical examples in the 
literature indicate, leads to highly accurate reconstructions with reasonable stability 
against errors in the far-field pattern. On the other hand, it should be noted that 
for the numerical implementation, an efficient forward solver is needed and good a 
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priori information is required in order to ensure convergence. On the theoretical side, 
the convergence of regularized Newton iterations for inverse obstacle scattering 
problems has not been completely settled, although some progress has been made 
through the work of Hohage [39] and Potthast [77]. 

Newton-type iterations can also be employed for the simultaneous determination 
of the boundary shape and the impedance function A in the impedance boundary 
condition (8) [58]. 


Decomposition Methods 


The main idea of decomposition methods is to break up the inverse obstacle 
scattering problem into two parts: The first part deals with the ill-posedness by 
constructing the scattered wave u* from its far-field pattern ug, and the second 
part deals with the nonlinearity by determining the unknown boundary 0D of the 
scatterer as the location where the boundary condition for the total field u' + u° is 
satisfied in a least-squares sense. In the potential method, for the first part, enough a 
priori information on the unknown scatterer D is assumed so one can place a closed 
surface I’ inside D. Then the scattered field u* is sought as a single-layer potential 


WO) = [ POI.» ds), x ERD. (77) 


with an unknown density g € L?(I). In this case the far-field pattern ug, has the 
representation 


7 1 _jkeey i 
Ugo (X) = ale fe *o(y)ds(y), XE s 


Given the far-field pattern ugg, the density g is now found by solving the integral 
equation of the first kind 


Soof = Uso (78) 


with the compact integral operator Soo : L?() > L?(S*) given by 


1 — ; 
(Sop 8) = Gf eMF pO) dy), FES? 


Due to the analytic kernel of S.o, the integral equation (78) is severely ill-posed. 
For a stable numerical solution of (78), Tikhonov regularization can be applied, i.e., 
the ill-posed equation (78) is replaced by 

Afa + S55ScoPu = S5oUloo (79) 


with some positive regularization parameter @ and the adjoint S%, of Soo. 
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Given an approximation of the scattered wave u%, obtained by inserting a solution 
Qu of (79) into the potential (77), the unknown boundary dD is then determined by 
requiring the sound-soft boundary condition 


uitu’ =0 ondD (80) 

to be satisfied in a least-squares sense, i.e., by minimizing the L? norm of the defect 
i s | 

J! + wale) (81) 


over a suitable set of admissible surfaces A. Instead of solving this minimization 
problem, one can also visualize 0D by color coding the values of the modulus |u| of 
the total field u ~ u' + us, ona sufficiently fine grid over some domain containing 
the scatterer. 

Clearly we can expect (78) to have a solution g € L?(I) if and only if ugg is the 
far field of a radiating solution to the Helmholtz equation in the exterior of with 
sufficiently smooth boundary values on I’. Hence, the solvability of (78) is related 
to the regularity properties of the scattered wave which in general cannot be known 
in advance for the unknown scatterer D. Nevertheless, it is possible to provide a 
solid theoretical foundation to the above procedure [22,53]. This is achieved by 
combining the minimization of the Tikhonov functional for (78) and the defect 
minimization for (81) into one cost functional 


2 
ae (82) 


2 2 i ‘ 
I| Soop _ Uco |l72(52) + aWlP\lp2¢r + Y \|u' + us, 


Here y > 0 denotes a coupling parameter which has to be chosen appropriately for 
the numerical implementation in order to make the two terms in (82) be of the same 
magnitude, for example, y = ||Uo0|| 12(52)/||U' lloo- 

Note that the potential approach can also be employed for the inverse problem 
to recover the impedance given the shape of the scatterer. In this case the far- 
field equation (78) is solved with I replaced by the known boundary 0D. After 
the density g is obtained, A can be determined in a least-squares sense from 
the impedance boundary condition (8) after evaluating the trace and the normal 
derivative of the single-layer potential (77) on dD. 

The point source method of Potthast [76] can also be interpreted as a decompo- 
sition method. Its motivation is based on Huygens’ principle from Theorem 4, i.e., 
the scattered field representation 


Aes i (9) (x,y) ds(v), x R'\D, (83) 
ap OV 


and the far-field representation 


u 


1 0 aa 
Uoo(%) = —7— / — (vye"** ds(y), RES’. (84) 
Jap Ov 
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For z € R3\D we choose a domain B, such that z ¢ B, and D C B, 
and approximate the point source ®(-,z) by a Herglotz wave function, i.e., a 
superposition of plane waves such that 


®(y.2) ® / eik49 (d)ds(d), y € Be (85) 
§2 


for some g, € L?(S7). Under the assumption that there does not exist a nontrivial 
solution to the Helmholtz equation in B, with homogeneous Dirichlet boundary 
condition on 0B,, the Herglotz wave functions are dense in H'/*(0B-) [23,30], and 
consequently the approximation (85) can be achieved uniformly with respect to y 
on compact subsets of B,. We can now insert (85) into (32) and use (33) to obtain 


u’(z) & 4a [. &(X)Uco(—X) ds(X) (86) 


as an approximation for the scattered wave u*. Knowing an approximation for the 
scattered wave, the boundary 0D can be found as above from the boundary condition 
(80). 

The approximation (85) can in practice be obtained by solving the ill-posed linear 
integral equation 


/ ek¥4 9 (d)ds(d) = ®(y,z), y € OB:, (87) 
S2 


via Tikhonov regularization and the Morozov discrepancy principle. Note that 
although the integral equation (87) is in general not solvable, the approximation 
property (86) is ensured through the above denseness result on Herglotz wave 
functions. 

An advantage of decomposition methods is that the separation of the ill- 
posedness and the nonlinearity is conceptually straightforward. A second and main 
advantage consists in the fact that their numerical implementation does not require a 
forward solver. As a disadvantage, as in the Newton method of the previous section, 
if we go beyond visualization of the level surfaces of |u| and proceed with the 
minimization, good a priori information on the unknown scatterer is needed for the 
iterative solution of the optimization problem. The accuracy of the reconstructions 
using decomposition methods is slightly inferior to that using Newton iterations. 


Iterative Methods Based on Huygens’ Principle 


We recall Huygens’ principle (83) and (84). In view of the sound-soft boundary 
condition, from (83), we conclude that 


0 
ul (x) = ‘| x (y) ®(x, y)ds(y), x € aD. (88) 
aD OV 
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Now we can interpret (84) and (88) as a system of two integral equations for the 
unknown boundary dD of the scatterer and the induced surface flux 


du 
g:=-— ond. 
dv 


It is convenient to call (84) the data equation since it contains the given far field for 
the inverse problem and (88) the field equation since it represents the boundary 
condition. Both equations are linear with respect to the flux and nonlinear with 
respect to the boundary. Equation (84) is severely ill-posed whereas (88) is only 
mildly ill-posed. 

Obviously there are three options for an iterative solution of (84) and (88). In 
a first method, given an approximation for the boundary 0D, one can solve the 
mildly ill-posed integral equation of the first kind (88) for g. Then, keeping ¢ fixed, 
Eq. (84) is linearized with respect to dD to update the boundary approximation. This 
approach has been proposed by Johansson and Sleeman [45]. In a second approach, 
following ideas first developed for the Laplace equation by Kress and Rundell [59], 
one also can solve the system (84) and (88) simultaneously for dD and g by Newton 
iterations, i.e., by linearizing both equations with respect to both unknowns. This 
idea has been analyzed by Ivanyshyn and Kress [42,43]. Whereas in the first method 
the burden of the ill-posedness and nonlinearity is put on one equation, in a third 
method, a more even distribution of the difficulties is obtained by reversing the 
roles of (84) and (88), i.e., by solving the severely ill-posed equation (84) for @ and 
then linearizing (88) to obtain the boundary update. With a slight modification, this 
approach may also be interpreted as a decomposition method since to some extent 
it separates the ill-posedness and the nonlinearity. It combines the decomposition 
method from the previous section “Decomposition Methods” with elements of 
Newton iterations from section “Newton Iterations in Inverse Obstacle Scattering”. 
Therefore, it has also been termed as a hybrid method and as such was analyzed by 
Kress and Serranho [57, 85]. 

For a more detailed description of these three methods, using the parameteriza- 
tion (70), we introduce the parameterized single-layer operator and far-field operator 
A, Age CCS") HLA (Sh > AS) by 


Ap We) = [HP® PEDWE) AH), FES, 

and 
ii ee a a 
Ago(p, W)(X) 7= ale w(~)ds(S), FES’. 


Then (84) and (88) can be written in the operator form 


Ago (P, W) = Uco (89) 
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and 


A(p,W) =—u' 0 p, (90) 


where we have incorporated the surface element into the density function via 


W(X) = J(X) e(p(X)) (91) 


with the Jacobian J of the mapping p. The linearization of these equations requires 
the Fréchet derivatives of the operators A and Ago with respect to p. These can be 
obtained by formally differentiating their kernels with respect to p, i.e., 


(A'(p, W)q)(X) = [. grad, ®(p(%), p(S))-[g(%) —g(P1 WO) ds(9), x € S*, 
and 


ik ot. Se S\ A a A rt 
(Abo(p, W)a) (8) = — Ze fe HPO LG WN d(9), xe S?, 


For fixed p, provided k is not a Dirichlet eigenvalue of the negative Laplacian 
in D, both in a Holder space setting A(p,-) : C°*(S?) + C!(S?) or in a 
Sobolev space setting A(p,-) : H7'/2(S?) — H'/?(S?), the operator A(p, -) 
is a homeomorphism [22]. In this case, given an approximation to the boundary 
parameterization p, the field equation (90) can be solved for the density y. Then, 
keeping y fixed, linearizing the data equation (89) with respect to p leads to the 
linear equation 


A’, | p. [A(p, JIG! © p) | ¢ = too — Aco | p. [A(p, -)I' i o p) | (92) 
-y —w 


for g to update the parameterization p via p + q. This procedure can be iterated. 
For fixed p the operator A’,(p,[A(p, -)]7"(u' © p)) has a smooth kernel 
and therefore is severely ill-posed. This requires stabilization, for example, via 
Tikhonov regularization. The following theorem ensures injectivity and dense range 
as prerequisites for Tikhonov regularization. We recall the form (74) introduced for 
uniqueness of the parameterization of the update and the corresponding linear space 
Ee ale *) of normal L? vector fields. 
Theorem 13. Assume that k* is not a Neumann eigenvalue of the negative Lapla- 
cian in D. Then the operator 


Al, (p. [A(p, I'l o p)) + Leotmai(S?) > L?(S?) 


is injective and has dense range. 
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One can relate this approach to the Newton iterations for the nonlinear equation 
(69) for the boundary to far-field operator of section “Newton Iterations in Inverse 
Obstacle Scattering”. In the case when k? is not a Dirichlet eigenvalue of the 
negative Laplacian in D, one can write 


F(p) = —Aco (p. [A(p. +) '(u' 0 p)) 


By the product and chain rule, this implies 


F'(p)q =—A, (p. [A(p. >)! © p)) ¢ 
+ Ago (p. [A(p. -)J'A'(p. [A(p, - (0 p)) a) (93) 
—Aoo (p, [A(p, «|! (gradu') 0 p)-q). 


Hence, we observe a relation between the above iterative scheme and the Newton 
iterations for the boundary to far-field map as expressed by the following theorem. 


Theorem 14. The iteration scheme given by (92) can be interpreted as Newton 
iterations for (69) with the derivative of F approximated by the first term in the 
representation (93). 


As to be expected from this close relation to Newton iterations for (69), the 
quality of the reconstructions via (92) can compete with those of Newton iterations 
with the benefit of reduced computational costs. 

The second approach for iteratively solving the system (89) and (90) consists in 
simultaneously linearizing both equations with respect to both unknowns. In this 
case, given approximations p and w both for the boundary parameterization and the 
density, the system of linear equations 


Ago(P.W)4 + Aoo(P, X) = —Aco(P. W) + too (94) 


and 


A'(p, W)q + ((gradu') 0 p)-q + A(p. x) = —-A(p,W) —u' 0 p (95) 


has to be solved for g and yx in order to obtain updates p + q for the boundary 
parameterization and yw + y for the density. This procedure again is iterated and 
coincides with Newton’s method for the system (89) and (90). 

For uniqueness reasons the updates must be restricted, for example, to normal 
fields of the form (74). Due to the smoothness of the kernels, both Eqs. (94) and (95) 
are severely ill-posed and require regularization with respect to both unknowns. In 
particular for the parameterization update, it is appropriate to incorporate penalties 
for Sobolev norms of g to guarantee smoothness of the boundary whereas for the 
density L? penalty terms on y are sufficient. 

The simultaneous iterations (94) and (95) again exhibit connections to the 
Newton iteration for (69). 
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Theorem 15. Assume that k? is not a Dirichlet eigenvalue of the negative Lapla- 
cian in D, and set y := —[A(p, -)|~'(u' 0 p). If q satisfies the linearized boundary 
to far-field equation (73), then q and 


x:= [AQ I (A4'(p. Wg + ((gradu') 0 p)-q) 


satisfy the linearized data and field equations (94) and (95). Conversely, if q and x 
satisfy (94) and (95), then q satisfies (73). 


Theorem 15 illustrates the difference between the iteration method based on 
(94) and (95) and the Newton iterations for (69). In general when performing (94) 
and (95) in the sequence of updates, the relation A(p, ¥) = —(u' o p) between 
the approximations p and yw for the parameterization and the density will not 
be satisfied. This observation also indicates a possibility to use (94) and (95) for 
implementing a Newton scheme for (69). It is only necessary to replace the update 
w + x for the density by —[A(p + q, -)]~'(u' 0 (p + q)), ie., at the expense of 
throwing away y and solving a boundary integral equation for a new density. For a 
numerical implementation and three-dimensional examples, we refer to [44]. 

In a third method, in order to evenly distribute the burden of the ill-posedness 
and the nonlinearity of the inverse obstacle scattering problem, instead of solving 
the field equation (90) for the density and then linearizing the data equation, one can 
also solve the severely ill-posed data equation (91) for the density and then linearize 
the mildly ill-posed field equation (92) to update the boundary. In this case, given an 
approximation for the boundary parameterization p, first the data equation (91) is 
solved for the density yw. Then, keeping w fixed, the field equation (92) is linearized 
to obtain the linear equation 


A'(p, W)q + ((gradu') 0 p)-q¢ = —A(p,W) —u' 0 p (96) 


for g to update the parameterization p via p + q. This procedure of alternatingly 
solving (91) and (96) can be iterated. To some extent this procedure mimics a 
decomposition method in the sense that it decomposes the inverse problem into 
a severely ill-posed linear problem and a nonlinear problem. 

The hybrid method suggested by Kress and Serranho [57, 85] can be considered 
as a slight modification of the above procedure. In this method, given an approxi- 
mation p for the parameterization of the boundary, the data equation (91) is solved 
for the density w via regularization. Injectivity and dense range of the operator 
Ago(p,:) : L*(S*) — L?(S7) are guaranteed provided k? is not a Dirichlet 
eigenvalue for the negative Laplacian in D [22]. Then one can define the single- 
layer potential 


w(x) = f OC, pW) AH) 
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and evaluate the boundary values of u := u' + u° and its derivatives on the surface 
represented by p via the jump relations. Finally an update p + q is found by 
linearizing the boundary condition uo (p + g) = 0, ie., by solving the linear 
equation 


uo p+ ((gradu) o p)-¢ =0 (97) 
for q. For uniqueness of the update representation, the simplest possibility is to allow 
only perturbations of the form (74). Then injectivity for the linear equation (97) can 


be established for the exact boundary. 
After introducing the operator 


(A(p.W)a) = f grad, OCC), pL) 4B) WH) AH) 


_ 1 ¥&) vP@))-¢@)] 
2 Ia) 


and observing the jump relations for the single-layer potential and (91), Eq. (97) can 
be rewritten as 


A(p,¥) q + (gradu!) 0 p)-q¢ = —A(p. W) —u' 0 p. (98) 


Comparing this with (96), we discover a relation between solving the data and 
field equation iteratively via (89) and (96) and the hybrid method of Kress and 
Serranho. In the hybrid method, the Fréchet derivative of A with respect to p is 
replaced by the operator A where one linearizes only with respect to the evaluation 
surface for the single-layer potential but not with respect to the integration surface. 
For the numerical implementation of the hybrid method and numerical examples in 
three dimensions, we refer to [86]. 

All three methods of this section can be applied to the Neumann boundary 
condition, the perfect conductor boundary condition, and to the impedance boundary 
condition in acoustics and electromagnetics. They also can be employed for the 
simultaneous reconstruction of the boundary shape and the impedance function A in 
the impedance boundary condition (8) [84]. 


Newton Iterations for the Inverse Medium Problem 


Analogously to the inverse obstacle scattering problem, we can reformulate the 
inverse medium problem as a nonlinear operator equation. To this end we define 
the far-field operator F :m +> Ugo that maps m := 1 —7 to the far-field pattern ugg 
for plane wave incidence u! (x) = e’***4, Since by Theorem 10 we know that m is 
uniquely determined by a knowledge of uoo(X,d) for all incident and observation 
directions ,d € S*, we interpret F as an operator from C(B) into L?(S? x $7) 
for a ball B that contains the unknown support of m. 


Inverse Scattering 683 


In view of the Lippmann—Schwinger equation (35) and the far-field representa- 
tion (36), we can write 


k hey 
(Fomye.d) =F fe moyuly.ddy, Rd eS% 09) 
B 
where u(-, d) is the unique solution of 
u(x,d)+ ef ®(x, y)m(y)u(y,d) dy =u'(x,d), x EB. (100) 
B 


From (100) it can be seen that the Fréchet derivative v, of u with respect to m (in 
direction q) satisfies the Lippmann—Schwinger equation 


vas d +e B(x, y)[m(y)vq(y,d)+q(y)u(y,d)|dy =0, xe B. (101) 


From this and (99), it follows that the Fréchet derivative of F is given by 


k2 aes 7 
(F'(m) q)(X,d) = 7 fe YIm(y)vg(v,d) +q(y)u(y, a] dy, &.d € S*, 


which coincides with the far-field pattern of the solution v,(-,d@) of (101). Hence, 
we have proven the following theorem. 


Theorem 16. The far-field mapping F : m +> Ugo is Fréchet differentiable. The 
derivative is given by 


F (tg = Vanes 
where Vq,o0 is the far-field pattern of the radiating solution vq to 
Av+k?nv = —k?ug_ in R?. (102) 
This characterization of the Fréchet derivative can be used to establish injectivity 
of F’(m). We now have all the prerequisites available for a regularized Newton 
iteration analogous to (76). 


A similar approach as that given above is also possible for the electromagnetic 
inverse medium problem. 


Least-Squares Methods for the Inverse Medium Problem 
In view of the Lippmann—Schwinger equation (35) and the far-field representation 


(36), the inverse medium problem is equivalent to solving the system consisting of 
the field equation 
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u(x,d)+ al B(x, y)m(y)u(y,d) dy = ul (x,d), x €B,d€S’, (103) 
B 


and the data equation 


k?2 


pees et*X-Ym(y)u(y, d) dy =Un(%,d), X,d€ S?, (104) 
An B 


where B is a ball containing the support of m. In principle one can first solve the ill- 
posed linear equation (104) to determine the source mu from the far-field pattern and 
then solve the nonlinear equation (103) to construct the contrast m. After defining 
the volume potential operator T : L?(B x S*) — L?(B x S”) and the far-field 
operator F : L*(B x S*) > L*(S? x S$?) by 


(Ty)ox.d) = -# f @Ge, yyv(y.d) dy. xe Bd € 8’, 
B 
and 
P KP ikea ° 2 
(Fv)(x, da) := =| ee *"“vy,d)dy, x,deS*, 
4n B 


we rewrite the field equation (103) as 
ul + Tmu =u (105) 
and the data equation (104) as 
Fmu = Ugo. (106) 
We can now define the cost function 
lui + Tmu — I| loo — Fmulli2¢52 


2 
ull 2(Bx82) x2) 


Lm, u) = (107) 


j 2 2 
I]u! lle2(Bx52) I|Uooll 52¢525¢52) 


and reformulate the inverse medium problem as the optimization problem to 
minimize jz over the contrast m € V and the fields u € W where V and W are 
appropriately chosen admissible sets. The weights in the cost function are chosen 
such that the two terms are of the same magnitude. 

This optimization problem is similar in structure to that used in (82) in connection 
with the decomposition method for the inverse obstacle scattering problem. How- 
ever, since by Theorem 10 all incident directions are required, the discrete versions 
of the optimization problem suffer from a large number of unknowns. Analogous 
to the two step approaches of sections “Decomposition Methods” and “Iterative 
Methods Based on Huygens’ Principle” for the inverse obstacle scattering problem, 
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one way to reduce the computational complexity is to treat the fields and the contrast 
separately, for example, by a modified conjugate gradient method as proposed by 
Kleinman and van den Berg [55]. In a modified version of this approach, van den 
Berg and Kleinman [89] transformed the Lippmann—Schwinger equation (105) into 
the equation 


mu + mTw = w (108) 
for the contrast sources w := mu, and instead of simultaneously updating the 
contrast m and the fields u, the contrast is updated together with the contrast source 


w. The cost function (107) is now changed to 


mu! + mTw — Wlliacy x52) (Moo — Finull2¢50.52) 


(Mm, Ww) 2= aT D 
|| e2(Bx52) || Uo Ih 2¢52552) 


The above approach for the acoustic inverse medium problem can be adapted to the 
case of electromagnetic waves. 


Born Approximation 


The Born approximation turns the inverse medium scattering problem into a linear 
problem and therefore is often employed in practical applications. In view of (36), 
for plane wave incidence, we have the linear integral equation 


2 
mae ek Gd) "Yn (y) dy = Ugo(%,d) %,d €S?. (109) 
An Jrr3 
Solving (109) for the unknown m corresponds to inverting the Fourier transform of 
m restricted to the ball of radius 2k centered at the origin, i.e., only incomplete data 
is available. This causes uniqueness ambiguities and leads to severe ill-posedness 
of the inversion. Thus, the ill-posedness which seemed to have disappeared through 
the inversion of the Fourier transform is back on stage. For details, we refer to [60]. 
A counterpart of the Born approximation in inverse obstacle scattering starts 
from the far field of the physical optics approximation (36) for a convex sound-soft 
scatterer D in the back scattering direction, i.e., 


1 0 


0° —d;d = = 
is 4x Jyy-d<o OV(Y) 


e2ikd-y ds(y). 


Analogously, replacing d by —d, we have 


I 
oo (d sd) = —5— / 


e 2ikd-y ds( ). 
(y)-d>0 dv(y) . 
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Combining the last two equations and using Green’s integral theorem, we find 


o xe dy = % {00(—d; d) + too(ds— =O\, deS*, (110) 
R3 


with the characteristic function x of the scatterer D. Equation (110) is known 
as the Bojarski identity. Hence, in the physical optics approximation, the Fourier 
transform has again to be inverted from incomplete data since the physical optics 
approximation is valid only for large wave numbers k. For details, we refer to [60]. 


Historical Remarks 


The boundary condition (75) was obtained by Roger [81] who first employed 
Newton-type iterations for the approximate solution of inverse obstacle scattering 
problems. Rigorous foundations for the Fréchet differentiability were given by 
Kirsch [47] in the sense of a domain derivative via variational methods and by 
Potthast [75] via boundary integral equation techniques. The potential method 
as a prototype of decomposition methods has been proposed by Kirsch and 
Kress [53]. The point source method has been suggested by Potthast [76]. The 
iterative methods based on Huygens’ principle were introduced by Johansson and 
Sleeman [45], by Ivanyshyn and Kress [43] (extending a method proposed by 
Kress and Rundel [59] from potential theory to acoustics), and by Kress [57] and 
Serranho [85]. The methods described in sections “Newton Iterations for the Inverse 
Medium Problem,” “Least Squares Methods for the Inverse Medium Problem,” and 
“Born Approximation” have been investigated by numerous researchers over the 
past 30 years. 


5 Qualitative Methods in Inverse Scattering 
The Far-Field Operator and Its Properties 


A different approach to solving inverse scattering problems than the use of iterative 
methods is the use of qualitative methods [7]. These methods have the advantage of 
requiring less a priori information than iterative methods (e.g., it is not necessary to 
know the topology of the scatterer or the boundary conditions satisfied by the total 
field) and in addition reduce a nonlinear problem to a linear problem. On the other 
hand, the implementation of such methods often requires more data than iterative 
methods do and in the case of a penetrable inhomogeneous medium only recovers 
the support of the scatterer together with some estimates on its material properties. 

We begin by considering the scattering problem for a sound-soft obstacle (58) 
and (59). The far-field operator F : L?(S*) — L?(S*) for this problem is defined 
by 
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(Fg)(X) i= i Ugo(X,d)g(d)ds(d), % € S’, (111) 


where Ugo is the far-field pattern associated with (58) and (59). By superposition, 
Fg is seen to be the far-field pattern corresponding to the Herglotz wave function 


ve(x) := 2 ed a(d)ds(d), x €R>, (112) 


as incident field. The function g € L?(S7) is known as the kernel of the Herglotz 
wave function. The far-field operator F is compact. It can also be shown that for the 
case of scattering by a sound-soft obstacle, the far-field operator is normal [7]. Of 
basic importance to us is the following theorem [22]. 


Theorem 17. The far-field operator F corresponding to (58) and (59) is injective 
with dense range if and only if there does not exist a Dirichlet eigenfunction for D 
which is a Herglotz wave function. 


Proof. The proof is based on the reciprocity relation (61). In particular, for the L? 
adjoint F* : L?(S*) > L?(S7), the reciprocity relation implies that 


F*¢ = RFR¢, (113) 


where R : L?(S”) — L?(S?) is defined by (Rg)(d) := g(—d). Hence, the operator 
F is injective if and only if its adjoint F'* is injective. Recalling that the denseness 
of the range of F is equivalent to the injectivity of F*, by (113), we need only 
to show the injectivity of F’. To this end, we note that Fg = 0 is equivalent to the 
existence of a Herglotz wave function vg with kernel g for which the far-field pattern 
of the corresponding scattered field v* is Vo9 = 0. By Rellich’s lemma this implies 
that v’ = 0 in R3\D and the boundary condition Vg + v° = 0 on dD now shows 
that ve = 0 on 0D. Since by hypothesis vg is not a Dirichlet eigenfunction, we can 
conclude that vy = 0 in D and hence g = 0. a 


We will now turn our attention to the far-field operator associated with the 
inhomogeneous medium problems (65) and (68). In both cases we again define 
the far-field operator by (111) where woo is now the far-field pattern corresponding 
to (65) or (68). We first consider Eq. (65) which corresponds to scattering by an 
inhomogeneous medium. The analogue of Theorem 17 is the following [22]. 


Theorem 18. The far-field operator F corresponding to (65) is injective with dense 


range if and only if there does not exist a solution v,w € L?(D),v—w € H?(D) of 
the interior transmission problem 


Av+k?yv=0 inD (114) 
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Aw+k?nw=0 inD (115) 
v=w onoD (116) 
ov ow 

— = — D 117 
dv Ov ane a 


such that v is a Herglotz wave function. Values of k > 0 for which there exists a 
nontrivial solution of (114)—-(117) are called transmission eigenvalues. 


A similar theorem holds for Eq.(68) which corresponds to scattering by an 
anisotropic medium where now (115) is replaced by 


V-AVw+k7nw =0 inD (118) 


and in (117) the normal derivative un is replaced by v- AVw. If the coefficients in 
(115) or (118) are real valued, then the far-field operator is normal. 


In the case of electromagnetic waves, the far-field operator becomes 
(Fela) = | Eoltd.g(d)dsd), Fe 82 (119) 
g2 


where now g € L?(S7), the space of square integrable tangential vector fields 
defined on S*, and Eo is the electric far-field pattern defined by (64). Theorems 
analogous to Theorems 17 and 18 are also valid in this case [22]. 


The Linear Sampling Method 
The linear sampling method is a non-iterative method for solving the inverse 
scattering problem that was first introduced by Colton and Kirsch [19] and Colton 
et al. [29]. To describe this method, we first consider the case of scattering by a 
sound-soft obstacle, i.e., (58) and (59), and assume that for every z € D, there 
exists a solution g = g(-,z) € L?(S”) to the far-field equation 

Fg = P0(-,2), (120) 


where 
1 beer a 
Cpe. aes. 
4a 


Since the right-hand side of (120) is the far-field pattern of the fundamental solution 
(13), it follows from Rellich’s lemma that 


i u’(x,d)g(d) ds(d) = ®(x, z) 
S2 
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for x € R3\D. From the boundary condition u = 0 on dD, we see that 
Ve(x) + O(x,z) =0 forx € aD, (121) 


where v, is the Herglotz wave function with kernel g. We can now conclude from 
(121) that vg becomes unbounded as z > x € 0D and hence 


lim “5% =o, 
Bure le( ) Il 22082) 
zeD 


i.e., OD is characterized by points z where the solution of (120) becomes unbounded. 

Unfortunately, in general, the far-field equation (120) does not have a solution 
nor does the above analysis say anything about what happens when z € R*\D. To 
address these issues, we first define the single-layer operator S : H~!/2(0D) > 
H'/2(aD) by 


(S9)(x) = i. (PR, 2) dS), € BD, 


define the Herglotz operator H : L?(0D) — H~'/?(aD) as the operator mapping g 
to the trace of the Herglotz wave function (112) on dD, and let F : H~'/*(@D) > 
L?(S) be defined by 


(Fe)(8) = / woe HY ds(y), eS? 


Then, using on the one hand the fact that Herglotz wave functions are dense in the 
space of solutions to the Helmholtz equation in D with respect to the norm in the 
Sobolev space H'!(D) and on the other the factorization of the far-field operator F 
as 


1 
F =—— FS""'H, 

4a 
one can prove the following result [7,52]. 
Theorem 19. Assume that k* is not a Dirichlet eigenvalue of the negative Lapla- 
cian for D, and let F be the far-field operator corresponding to (58) and (59). 
Then: 
1. Forz € D anda givene > 0, there exists g..< € L?(S7) such that 

|| Fez — Doo(-, 2) Il z2(52) <€ 


and the corresponding Herglotz wave function vg. . converges to a solution of 
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Au+tk?u=0 inD 
u=—®(-,z) ondD 


in H'(D) ase +0. 
2. Forz€ R?\D and a given € > 0, every g.< € L?(S”) that satisfies 


| F226 — Poo(-,Z)||12(82) < € 
is such that lime-+9 ||Vg..\|#1(p) = ©: 


We note that the difference between cases (1) and (2) of this theorem is that for 
z € D, the far-field pattern ®,,(-, z) is in the range of F, whereas for z € R*\D 
this is no longer true. The linear sampling method is based on attempting to 
compute the function g,, in the above theorem by using Tikhonov regularization 
to solve Fg = ®,(-,z). In particular, one expects that the regularized solution 
will be relatively smaller for z in D than z in R* \D, and this behavior can be 
visualized by color coding the values of the regularized solution on a grid over some 
domain containing D. A more precise statement of this observation will be made 
in the next section after we have discussed the factorization method for solving 
the inverse scattering problem. Further discussion of why linear sampling works if 
regularization methods are used to solve (120) can be found in [2, 3]. In addition 
to the inverse scattering problems (58) and (59), it is also possible to treat mixed 
boundary value problems as well as scattering by both isotropic and anisotropic 
inhomogeneous media where in the latter case we must assume that k is not a 
transmission eigenvalue. For full details we refer the reader to [7]. Note that in each 
case, it is not necessary to know the material properties of the scatterer in order to 
determine the support of the scatterer from a knowledge of the far-field pattern via 
solving the far-field equation Fg = ®o0(-, z). 

The linear sampling method can also be extended to the case of electromagnetic 
waves where the far-field equation (120) is now replaced by 


[Bx Gd. gd) 491d) = BsaolS24). 


where Eo (X, d, p) is the electric far-field pattern corresponding to the incident field 
(50), g € L?(S =). and E. oo is the electric far-field pattern of the electric dipole 


E.(x,z,q) = ; curl, curl, g®(x,z), He(x,z,q) := curl, g®(x, z). (122) 


Full details can be found in the lecture notes [12]. 

We close this section by briefly describing a version of the linear sampling 
method based on the reciprocity gap functional which is applicable to objects sit- 
uated in a piecewise homogeneous background medium. Assume that an unknown 
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scattering object is embedded in a portion B of a piecewise inhomogeneous medium 
where the index of refraction is constant with wave number k. Let Bp C B bea 
domain in B having a smooth boundary 0Bo such that the scattering obstacle D 
satisfies D C Bo, and let v be the unit outward normal to 0By. We now define the 
reciprocity gap functional by 


dv Ou 
Ru, v) ‘= I, (« a x) ds, 


where u and vy are solutions of the Helmholtz equation in Bo\D and u,v € 
C!(Bo\D). In particular, we want u to be the total field due to a point source situated 
at x9 € B\Bo and v = Vg to be a Herglotz wave function with kernel g. We then 
consider the integral equation 


R(u, vg) = R(u, ®.), 


where ®, := (-, z) is the fundamental solution (13) and u = u(-, x9) where Xo is 
now assumed to be on a smooth surface C in B \Bo that is homotopic to 0Bp. If D is 
a sound-soft obstacle, we assume that k? is not a Dirichlet eigenvalue of the negative 
Laplacian in D, and if D is an isotropic inhomogeneous medium, we assume that k 
is not a transmission eigenvalue. We then have the following theorem [17]. 


Theorem 20. Assume that the above assumptions on D are satisfied. Then: 


1. Ifz € D, then there exists a sequence {gy} in L?(S*) such that 


lim R(u,vg,) = R(u, ®,), x0 EC, 
noo 


and Vz, converges in L?(D). 
2. If z € Bo\D, then for every sequence {g,} in L?(S7) such that 


lim R(u, vg,) = R(u, ®,), x0 EC, 
noo 


we have that limn—oo ||V¢, |lL2(~p) = ©: 


In particular, Theorem 20 provides a method for determining D from a knowl- 
edge of the Cauchy data of uv on 0Bo in a manner analogous to that of the linear 
sampling method. Numerical examples using this method can be found in [17]. 
The extension of Theorem 20 to the Maxwell equations, together with numerical 
examples, can be found in [13]. 
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The Factorization Method 


The linear sampling method is complicated by the fact that in general ®go(-, z) is 
not in the range of the far-field operator F for either z € D or z € R° \D. For the 
case of acoustic waves when F is normal (e.g., the scattering problem corresponding 
to (58) and (59) or (65) for 1 real valued), the problem was resolved by Kirsch in 
[48,49] who proposed replacing the far field equation Fg = ®,o(-, z) by 


(F*F)'4¢ = ®o0(-,2), (123) 


where F* is again the adjoint of F in L?(S7). In particular, if G : H'/*(@D) > 
L?(S”) is defined by Gf = Voo where Voo is the far field pattern of the solution to 
the radiating exterior Dirichlet problem (see Theorem 3) with boundary data f € 
L?(dD), then the following theorem is valid [48]. 


Theorem 21. Assume that k* is not a Dirichlet eigenvalue of the negative Lapla- 
cian for D. Then the ranges of G : H'/?(0D) — L?(S?) and (F*F)'/* : 
L?(S*) + L?(S?) coincide. 


A result analogous to Theorem 21 is also valid for the scattering problem 
corresponding to (65) for 1 real valued where we now must assume the k is not an 
interior transmission eigenvalue [49]. Note that Theorem 21 provides an alternate 
method to the linear sampling method for solving the inverse scattering problem 
corresponding to the scattering of acoustic waves by a sound-soft obstacle. This 
follows from the fact that ®.o(-, z) is in the range of G if and only if z € D,i-e., 
Eq. (123) is solvable if and only if z € D. This is an advantage over the linear 
sampling method since if (123) is solved by using Tikhonov regularization, then as 
the noise level on uo, tends to zero, the norm of the regularized solution remains 
bounded if and only if z ¢ D. A similar statement cannot be made if regularization 
methods are used to solve Fg = ©®.(-,z). However, using Theorem 21, the 
following theorem has been established by Arens and Lechleiter [3] (see also [52]). 


Theorem 22. Let F be the far-field operator associated with the scattering prob- 
lems (58) and (59), and assume that k? is not a Dirichlet eigenvalue of the 
negative Laplacian for D. For z € D, let g, € L?(S7) be the solution of 
(F*F)'/4g. = ®4(-,2), and for every z € R? ande > 0, let g,¢ be the solution 
of Fg = ®oo(-, z) obtained by Tikhonov regularization, i.e., the unique solution of 
eg + F* Fg = F*®,. Then the following statements are valid: 


1. Let vg... be the Herglotz wave function with kernel g,-. Then for every z € D, 


the limit lim,_,o Vg, .(Z) exists. Furthermore, there exists c > 0, depending only 
on F, such that for every z € D we have that 


2 . 2 
ellSzllz2¢s2yl = lim |v2..<(2)| < Il Zl 2,52): 


2. Forz ¢ D we have that lime-+0 Vg, .(Z) = ©. 
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Using Theorem 21 to solve the inverse scattering problem associated with the 
scattering problems (58) and (59) is called the factorization method. This method 
has been extended to a wide variety of scattering problems for both acoustic and 
electromagnetic waves, and for details we refer the reader to [52]. Since this 
method and its generalizations are fully discussed in the chapter in this handbook 
on sampling methods, we will not pursue the topic further here. A drawback of 
both the linear sampling method and the factorization method is the large amount of 
data needed for the inversion procedure. In particular, although the linear sampling 
method can be applied for limited aperture far-field data, one still needs multistatic 
data defined on an open subset of $7. 


Lower Bounds for the Surface Impedance 


One of the advantages that the linear sampling method has over other qualitative 
methods in inverse scattering theory is that the far-field equation can not only be 
used to determine the support of the scatterer but in some circumstances can also 
be used to obtain lower bounds on the constitutive parameters of the scattering 
object. In this section we will consider two such problems: the determination of the 
surface impedance of a partially coated object and the determination of the index of 
refraction of a non-absorbing scatterer. In the first case, we will need to consider a 
mixed boundary value problem for the Helmholtz equation, whereas in the second 
case, we will need to investigate the spectral properties of the interior transmission 
problem introduced in Theorem 18 of the previous section. 

Mixed boundary value problems typically model the scattering by objects that 
are coated by a thin layer of material on part of the boundary. In the study of inverse 
problems for partially coated obstacles, it is important to mention that, in general, 
it is not known a priori whether or not the scattering object is coated and if so 
what the extent of the coating is. We will focus our attention in this section on the 
special case when on the coated part of the boundary, the total field satisfies an 
impedance boundary condition and on the remaining part of the boundary, the total 
field (or the tangential component in the case of electromagnetic waves) vanishes. 
This corresponds to the case when a perfect conductor is partially coated by a thin 
dielectric layer. For other mixed boundary value problems in scattering theory and 
their associated inverse problems, we refer the reader to [7] and the references 
contained therein. 

Let D C R? be as described in Sect. 1, and let 9D be dissected as DD = Tp U 
I UT, where Ip and I; are disjoint, relatively open subsets of 0D having I as 
their common boundary. Let A € Loo (IT) be such that A(x) > Ao > O for all 
x € T;. We consider the scattering problem for the Helmholtz equation (58) where 
u =u! + w’ satisfies the boundary condition 


u=0 only, 
a 124 
5p tT Au=0 on Ty, ee 
Vv 
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u(x) = e***4, and uv’ is a radiating solution. It can be shown that this direct 
scattering problem has a unique solution in Aig (R? \D) [7]. We again define the 
far-field operator by (111) where ugg is now the far-field pattern corresponding to 
the boundary condition (124). 

In [7] it is shown that there exists a unique solution u, € H'(D) of the interior 
mixed boundary value problem 


Au.+k?u,=0 inD (125) 
u,+ ®(-,z) =O onTp (126) 
a 
ce (uz + ®(-,z)) + iA (uz + O(-,z)) =O only (127) 


for z € D where @ is the fundamental solution to the Helmholtz equation. Then, if 
®oo(-, Z) is the far-field pattern of ®(-, z), we have the following theorem [7]. 


Theorem 23. Let « > 0, z € D, and u, be the unique solution of (125-127). Then 
there exists a Herglotz wave function vg, . with kernel g.,. € L”(S”) such that 


lluz — Vg. ellHi(p) XS €- 
Moreover, there exists a positive constant c independent of € such that 
| Fez — Pool-, 2I\l12(52) S ce. 


We can now use Green’s formula to show that [6] 


k 
I Alu, + ®(-,2)|? ds = ——— — Imu,(2). 
aD An 


From this we immediately deduce the inequality 


—k/4x —Imu;(z) 
Uz 7 D(., alee 


IA Ilzeocry) = (128) 


How is the inequality (128) of practical use? To evaluate the right-hand side of 
(128), we need to know 0D and u;. Both are determined by solving the far field 
equation Fg = ®,.(-,z) using Tikhonov regularization and then using the linear 
sampling method to determine dD and the regularized solution g € L?(S*) to 
construct the Herglotz wave function vz. By Theorem 23 we expect that v, is an 
approximation to u,. However, at this time, there is no analogue of Theorem 22 for 
the mixed boundary value problem, and hence this is not guaranteed. Nevertheless 
in all numerical experiments to date, this approximation appears to be remarkably 
accurate and thus allows us to obtain a lower bound for ||A || ,-o(r,) via (128). 


Inverse Scattering 695 


The corresponding scattering problem for the Maxwell equations is to find a 
solution E = E' + E* to (62) satisfying the mixed boundary condition 


vxE=0 onTIp 
vxcurlE —idA(vx E)xv=0 only, (129) 
where E’ is the plane wave (50) and E° is radiating. The existence of a unique 
solution E in an appropriate Sobolev space is shown in [11]. We again define the far- 
field operator by (119) where E, is now the electric far-field pattern corresponding 


to (129). Analogous to (125-127) we now have the interior mixed boundary value 
problem 


curlcurl E,—k?E,=0 inD (130) 

vx [E,+ Ee(-,z,q)] =0 onTp (131) 

v x curl[E, + E.(-,z,q)] —iA[v x (EF, + E.(-,z,qg))] =0 only, (132) 
where z € D and E£, is the electric dipole defined by (122). The existence of a 


unique solution to (130-132) in an appropriate Sobolev space is established in [11]. 
From the analysis in [6], we have the inequality 


—k*|q/?/60 + k Re(q- E.) 


(133) 
|E, a Ee( *,%, Dia) 


A llzoowry) = 


analogous to (128) for the Helmholtz equation. For numerical examples using (133), 
we refer the reader to [26]. 

Similar inequalities as those derived above for the impedance boundary value 
problem can also be obtained for the conductive boundary value problem, i.e., the 
case when a dielectric is partially coated by a thin, highly conducting layer [7, 26]. 


Transmission Eigenvalues 


We have previously encountered transmission eigenvalues in Theorem 18 where 
they were connected with the injectivity and dense range of the far-field operator. In 
this section we shall examine transmission eigenvalues and the interior transmission 
problem in more detail. This investigation is particularly relevant to the inverse 
scattering problem since transmission eigenvalues can be determined from the far 
field pattern [10] and, as will be seen, can be used to obtain lower bounds for the 
index of refraction. 

We begin by considering the interior transmission problem (114-117) from 
Theorem 18 and will be concerned with the existence and countability of transmis- 
sion eigenvalues. The existence of transmission eigenvalues was first established 
by Paivarinta and Sylvester [73], and their results were strengthened by Cakoni 
et al. [14]. 
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Theorem 24, Assume that n is real valued such that n(x) > 1 forall x € D or 
0 < n(x) < 1 forall x € D. Then there exist an infinite number of transmission 
eigenvalues. 


We note that it can be shown that as sup,ep |n(x) — 1| — 0, then the 
first transmission eigenvalue tends to infinity, i.e., in the Born approximation 
transmission, eigenvalues do not exist [28]. 

Similar results as in Theorem 24 can be obtained for an anisotropic medium and 
for the Maxwell equations [14]. 

By Theorem 24 the existence of transmission eigenvalues is established. It can 
also be shown that the set of transmission eigenvalues is discrete [15, 20, 50, 83]. 
The following theorem [28] establishes a lower bound for the first transmission 
eigenvalue which is reminiscent of the famous Faber—Krahn inequality for the first 
Dirichlet eigenvalue for the negative Laplacian (which we denote by 41). 


Theorem 25. Assume that n(x) > 1 for x € D, and let kj > 0 be the first 
transmission eigenvalue for the interior transmission problem (114-117). Then 


k2 = A\(D) 


"= sup n(x) 
xeED 


Theorem 25 has been generalized to the case of anisotropic media and the Maxwell 
equations [8]. 

Finally, in the case of the interior transmission problem (114-117) where there 
are cavities in D, i.e., regions Dp C D where n(x) = | for x € Do, it can be shown 
that transmission eigenvalues exist and form a discrete set and the first transmission 
eigenvalue k, satisfies [9] 


'= sup n(x) 
xED\Do 


Note that since in each of the above cases D can be determined by the linear 
sampling method, A, (D) is known, and hence given ky, the above inequalities yield 
a lower bound for the supremum of the index of refraction. 


Historical Remarks 


The use of qualitative methods to solve inverse scattering problems began with the 
1996 paper of Colton and Kirsch [19] and the 1997 paper of Colton et al. [29]. 
These papers were in turn motivated by the dual space method of Colton and Monk 
developed in [24,25]. Both [19] and [29] were concerned with the case of scattering 
of acoustic waves. The extension of the linear sampling method to electromagnetic 
waves was first outlined by Kress [56] and then discussed in more detail by Colton 
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et al. [18] and Haddar and Monk [34]. The factorization method was introduced 
in 1998 and 1999 by Kirsch [48, 49] for acoustic scattering problems. Attempts to 
extend the factorization method to the case of electromagnetic waves have been 
only partly successful. In particular, the factorization method for the scattering of 
electromagnetic waves by a perfect conductor remains an open question. 

In addition to the linear sampling and factorization methods, there have been 
a number of other qualitative methods developed primarily by Ikehata and Potthast 
and their coworkers. Although space is too short to discuss these alternate qualitative 
methods in this survey, we refer the reader to [52,78] for details and references. 

The countability of transmission eigenvalues for acoustic waves was established 
by Colton et al. [20] and Rynne and Sleeman [83] and for the Maxwell equations by 
Cakoni and Haddar [15] and Kirsch [50]. The existence of transmission eigenvalues 
for acoustic waves was first given by Pdivarinta and Sylvester [73] for the isotropic 
case and for the anisotropic case by Cakoni and Haddar [16] and Kirsch [51] who 
also established the existence of transmission eigenvalues for Maxwell’s equations. 
These results were subsequently improved by Cakoni et al. [14]. Inequalities for the 
first transmission eigenvalues were first obtained by Colton et al. [28] and Cakoni 
et al. [8,9]. 
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(or conductivity and permittivity) of the interior of a body from knowledge of 
electrical stimulation and measurements on its surface. This problem is also 
known as the inverse conductivity problem and its mathematical formulation is 
due to A. P. Calderén, who wrote in 1980, the first mathematical formulation 
of the problem, “On an inverse boundary value problem.” EIT has interesting 
applications in fields such as medical imaging (to detect air and fluid flows in the 
heart and lungs and imaging of the breast and brain) and geophysics (detection of 
conductive mineral ores and the presence of ground water). It is well known that 
this problem is severely ill-posed, and thus this chapter is devoted to the study of 
the uniqueness, stability, and reconstruction of the conductivity from boundary 
measurements. A detailed distinction between the isotropic and anisotropic case 
is made, pointing out the major difficulties with the anisotropic case. The issues 
of global and local measurements are studied, noting that local measurements are 
more appropriate for practical applications such as screening for breast cancer. 


1 Introduction 


Electrical impedance tomography (EIT) is the recovery of the conductivity (or 
conductivity and permittivity) of the interior of a body from a knowledge of currents 
and voltages applied to its surface. In geophysics, where the method is used in 
prospecting and archaeology, it is known as electrical resistivity tomography. In 
industrial process tomography it is known as electrical resistance tomography or 
electrical capacitance tomography. In medical imaging, when at the time of writing 
it is still an experimental technique rather than routine clinical practice, it is called 
EIT. A very similar technique is used by weakly electric fish to navigate and 
locate prey and in this context it is called electrosensing. An example of a medical 
application of EIT is given in Fig. 1, which shows a 10-day-old healthy neonate 
breathing spontaneously and lying in the prone position with the head turned to the 
left. Sixteen EIT electrodes are placed in a transverse plane around the chest, and 
EIT data acquired with the Goe MF-II system. This child was a subject in a study 
which used EIT to examine patterns of breathing in neonates and the relationship 
to body posture [57]. In this study, EIT was able to show, for the first time, that, in 
a prone position, the lung on the opposite side (contralateral) to the face receives 
significantly larger air flows. 

The simplest mathematical formulation of inverse problem of EIT can be stated 
as follows. Let Q be a conducting body described by a bounded domain in R", 
n > 2, with electrical conductivity a bounded and positive function y(x) (later 
complex y will be considered). In the absence of internal sources, the electrostatic 
potential uw in Q is governed by the elliptic partial differential equation 


Lyu:= V-yVu=0 in Q. (1) 
It is natural to consider the weak formulation of (1) in which u € H!(Q) isa 


weak solution to (1). Given a potential ¢ ¢ H'/?(0Q) on the boundary, the induced 
potential wu € H'(Q) solves the Dirichlet problem 
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Fig. 1 10-day-old 
spontaneously breathing 
neonate lying in the prone 
position with the head turned 
to the left. Sixteen medical 
grade Ag/AgCl electrodes 
were placed in a transverse 
plane and connected to a Geo 
MF-II EIT system [57] 


= in Q, (2) 


ulag = gd. 


The currents and voltages measurements taken on the surface of Q, dQ, are 
given by the so-called Dirichlet-to-Neumann map (associated with y) or voltage- 
to-current map 


Ay : ulag € H'/?(8Q) — yo e H~'/?(aQ). 
v 


Here, v denotes the unit outer normal to dQ and the restriction to the boundary 
is considered in the sense of the trace theorem on Sobolev spaces. Here, dQ must 
be at least Lipschitz continuous and y € L°(Q2) with ess infRe y = m > 0. 

The forward problem under consideration is the map y € Dy +> A,, where 
Dm = {y € L®(Q)lessinfy > m} The inverse problem for complete data is then 
the recovery of y from A,. As is usual in inverse problems, consideration will be 
given to questions of (1) uniqueness of solution (or from a practical point of view 
sufficiency of data), (2) stability/instability with respect to errors in the data, and 
(3) practical algorithms for reconstruction. It is also worth pointing out to the reader 
who is not very familiar with EIT the well-known fact that the behavior of materials 
under the influence of external electric fields is determined not only by the electrical 
conductivity y but also by the electric permittivity ¢ so that the determination of 
the complex valued function y(x, @) = o(x) + iwe(x) would be the more general 
and realistic problem, where i = /—1 and @ is the frequency. The simple case 
where w = 0 will be treated in this work. For a description of the formulation of the 
inverse problem for the complex case, refer for example to [20]. Before addressing 
questions (1)—(3) mentioned above, it is interesting to consider how the problem 
arises in practice. 
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Measurement Systems and Physical Derivation 


For the case of direct current, that is the voltage applied is independent of time, the 
derivation is simple. Of course here 2 C R’. First suppose that it is possible to 
apply an arbitrary voltage ¢ €¢ H'/?(Q) to the surface. It is typical to assume that 
the exterior R*\Q is an electrical insulator. An electric potential (voltage) u results 
in the interior and the current J that flows satisfies the continuum Ohm’s law J = 
—y Vu; the absence of current sources in the interior is expressed by the continuum 
version of Kirchoff’s law V-J = 0 which together result in (1). The boundary 
conditions are controlled or measured using a system of conducting electrodes 
which are typically applied to the surface of the object. In some applications, 
especially geophysical, these may be spikes that penetrate the object, but it is 
common to model these as points on the surface. Most EIT systems are designed to 
apply a known current on (possibly a subset) or electrodes and measure the voltage 
that results on electrodes (again possibly a subset, in some cases disjoint from those 
carrying a nonzero current). In other cases it is a predetermined voltage applied 
to electrodes and the current measured; there being practical reasons determined 
by electronics or safety for choosing one over the other. In medical EIT applying 
known currents and measuring voltages is typical. One reason for this is the desire 
to limit the maximum current for safety reasons. In practice the circuit that delivers a 
predetermined current can only do so while the voltage required to do that is within 
a certain range so both maximum current and voltage are limited. For an electrode 
(let us say indexed by £) not modeled by a point but covering a region Ey, C dQ the 
current to that electrode is the integral 


n= ainda (3) 
Ey 


Away from electrodes, 


du 


L 
on = 0, on 2\ |_) Ee (4) 


£=1 


as the air surrounding the object is an insulator. On the conducting electrode, u|z, = 
V; a constant, or as a differential condition 


L 
vx Vu=0 on dQ\(_) Ev. (5) 
t=1 


Taken together, (3)-(5) are called the shunt model. This ideal of a perfectly 
conducting electrode is of course only an approximation. Note that while the 
condition u € H'!(Q) is a sensible condition, which ensures finite total dissipated 
power, it is not sufficient to ensure (3) is well defined. Indeed for smooth y and 
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smooth d£;, the condition results in a square root singularity in the current density 
on the electrode. A more realistic model of electrodes is given later. 

It is more common to use alternating current in geophysical and process 
monitoring applications, and essential in medical applications. Specifically the 
direction of the current must be reversed within a sufficiently short time to avoid 
electrochemical effects. This also means that the time average of the applied current 
should be zero. In medical applications, current in one direction for sufficient 
duration would result in transport of ions, and one of the effects of this can be 
stimulation of nerves. It would also degrade electrode behavior due to charge build 
up and ionic changes in the electrode. As a general rule higher levels of current 
and voltage are considered safer at higher temporal frequencies. The simplest EIT 
system therefore operates at a fixed frequency using an oscillator or digital signal 
processing to produce a sinusoidal current. Measurements are then taken of the 
magnitude, or in some cases the components that are in phase and 7/2 out of phase 
with the original sine wave. When an EIT system switches from stimulating one 
set of electrodes to the next set in a stimululation pattern, the measurements adapt 
to the new pattern over a finite time, and typical EIT systems are designed to start 
measuring after this transient term has decayed so as to be negligible. 

In geophysics a technique that is complementary to EIT called induced polar- 
ization tomography IPT is used to find polarizable minerals. In effect this uses a 
square wave pulse and measures the transient response [85]. In process tomography 
a technique known as electrical capacitance tomography is designed for imaging 
insulating materials with different dielectric permittivities, for example oil and gas 
in a pipe [60, 101]. Again square waves or pulses are used. 

In medical and geophysical problems the response of the materials may vary with 
frequency. For example in a biological cell higher frequency current might penetrate 
a largely capacitive membrane and so be influenced by the internal structures 
of the cell, while lower frequency currents pass around the cell. This has led 
to electrical impedance tomography spectroscopy (EITS) [47], and in geophysics 
spectral induced polarization tomography (SIPT) [85]. The spectral response can 
be established either by using multiple sinusoidal frequencies or by sampling the 
transient response to a pulse. 

Our starting point for the case of alternating current is the time harmonic 
Maxwell equations at a fixed angular frequency w. Here it is assumed that the 
transient components of all fields are negligible and represent the time harmonic 
electric and magnetics vector fields using the complex representation F(x,f) = 
Re (Fexp(iwt)), yielding 


VxE= —ioB, (6) 
VxH=J+ioD. (7) 
The electric and magnetic fields E and H are related to the current density J, 


electric displacement D, and magnetic flux B by the material properties conductivity 
Oo, permittivity €, and permeability yw by 
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J=oE, D=ck, B= pH. (8) 


The fields E and H are evaluated on directed curves, while the “fluxes” J, D and 
B on surfaces. In biomedical applications one can typically take wz to be constant 
and to be the same inside the body as outside in air. In non-destructive testing and 
geophysical applications there may well be materials with differing permeability. 
Here (8) assumes linear relations. For example the first is the continuum Ohm’s 
law. Here, the material properties may be frequency dependent. This dispersion 
is important in EIS and SIPT. For the moment a first approximation is to assume 
isotropy (so that the material properties are scalars). 

There are many inverse problems governed by time harmonic Maxwell’s equa- 
tions.. These occur at large values of w and include optical and microwave 
tomographic techniques and scattering problems such as radar which are not 
discussed in this chapter. There are also systems where the fields arise from 
alternating current in a coil, and measurements are made either with electrodes or 
with other coils. Mutual (or magnetic) induction tomography (MIT) falls into this 
category and has been tried in medical and process monitoring applications [46]. 
In these cases the eddy current approximation [9] to Maxwell’s equations is used. 
While for direct current EIT (i.e., ERT) the object is assumed surrounded by an 
insulator, in MIT one must account for the magnetic fields in the surrounding space, 
there being no magnetic “shielding.” 

The assumptions used to justify the usual mathematical model of EIT make it 
distinct from many other inverse problems for Maxwell’s equations. Given 


Assumption 1. Transients components of all fields are negligible. 


This assumption simply means that a sufficient “settling time” has been given 
before making measurements. 

The interest in relatively low frequencies, where magnetic effects can be 
neglected, translates into two assumptions 


Assumption 2. w/e is small compared with the size of Q. 
This means that the wavelength of propagating waves in the material is large. A 


measurement accuracy of 2~!* = 1/4096 is ambitious at higher frequencies means 
that for wave effects to be negligible 


4095 
dw /el < cos! 7096’ (9) 


where d is the diameter of the body. Taking the relative permittivity to be 10 and 
R = 0.3 m gives a maximum frequency of | MHz. 


Assumption 3. ,/woj./2 is small compared with the size of Q. 
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Fig. 2. A system of electrodes used for chest EIT at Oxford Brookes University. The positions 
of the electrodes were measured manually with a tape measure and the cross-sectional shape was 
also determined by manual measurements. These electrodes have a disk of jell containing silver 
chloride solution that makes contact with the skin. Each electrode was attached to the EIT system 
by a screened lead, not shown in this picture for clarity 


The quantity 


6 = ,f — (10) 


is known as the skin depth. For a frequency of 10kHz and a conductivity of 
0.5 Sm! typical in medical applications, the skin depth is 7 m. In geophysics lower 
frequencies are typical but length scales are larger. In a conducting cylinder the 
electric field decays with distance r from the boundary at a rate e~"/* due to the 
opposing magnetic field. At EIT frequencies this simple example suggests that 
accurate forward modeling of EIT should take account of this effect although it 
is currently not thought to be a dominant source of error. 

The effect of Assumptions 2 and 3 combined together is that it is reasonable to 
neglect V x E in Maxwell’s equations resulting in the standard equation for complex 
EIT 


V-(o +iwe)Vu = 0. (11) 


Here the expression y = o + iwe is called complex conductivity, or logically 
the admittivity, while 1/o is called resistivity and the rarely-used complex 1/y 
impedivity. A scaling argument is given for the approximation (11) in [31], and 
numerical checks on the validity of the approximation in [37, 104] (Fig. 2). 

It is often not so explicitly stated but while in the direct current case one can 
neglect the conductivity of the air surrounding the body, for the alternating current 
case the electrodes are coupled capacitively and, while o can be assumed to be 
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Fig. 3 A cross section 
through a typical ECT sensor 
around a pipe (internal wall) 
showing the external screen 
with radial screens designed 
to reduce the external 
capacitive coupling between 
electrodes 


electrode 


radial earthed 
screen 


‘earthed Screen 


internal wall 


zero for air, the permittivity of any material is no smaller than that of a vacuum 
€) = 8.85 x 10~!?, although dry air approaches that value. One requires then 


Assumption 4. we in the exterior is negligible compared to |o +iqe| in the interior. 


For example, with a conductivity or 0.2Sm7!, the magnitude of the exterior 
admittivity reaches 2~'* of that value for a frequency of 0.88 MHz. For a more 
detailed calculation the capacitance between the electrodes externally could be 
compared with the impedance between electrodes. In ECT frequencies above 1 MHz 
are used and the exterior capacitance cannot be neglected. Indeed an exterior 
grounded shield is used so that the exterior capacitive coupling is not affected by 
the surroundings (see Fig. 3). 


The Concentric Anomaly: A Simple Example 


A simple example helps us to understand the instability in the inverse conductivity 
problem. Let Q be the unit disk in R* with polar coordinates (r, 9) and consider a 
concentric anomaly in the conductivity of radius p < 1 


a1, |x| <p 


12 
ao, p< |x| <1. = 


y(x) = 


From separation of variables, matching Dirichlet and Neumann boundary condi- 
tions at |x| = p, forn € Z 
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i ppl elf 


ind __ 
A,ye"’ = |n| T= ppm 


: (13) 


where jt = (a, — do)/(a, + do). From this one sees the effect of the Dirichlet 
to Neumann map on the complex Fourier series, and the effect on the real Fourier 
series is easily deduced. This example was considered in [64] as an example of the 
eigenvalues and eigenfunctions of A,, and also by [2] as an example of instability. 
Thus ||y — do||z-0(@) = |a1 — ao| independently of p and yet A, — Aggy in the 
operator norm. Hence if an inverse map A, +> y exists, it cannot be continuous 
in this topology. Similar arguments can be used to show instability of inversion in 
other norms. 

This example reveals many other features of the more general problem. For 
example experimentally one observes saturation: for an object placed away from 
the boundary, changes in the conductivity of an object with a conductivity close 
to the background are fairly easily detected, but for an object of very high or low 
conductivity further changes in conductivity of that object have little effect. This 
saturation effect was explored for offset circular objects (using conformal mappings) 
by Seagar [98]. This is also an illustration of the nonlinearity of y > A,. One can 
also see in this example that smaller objects (with the same conductivity) produce 
smaller changes in measured data as one might expect. 

On the unit circle S$! one can define an equivalent norm on the Sobolev space 
H(S') (see definitions in the section “The Neumann-to-Dirichlet Map”) by 


2 


[o,@) CO 
) c,mre"?|| = ) ns, (14) 
n=—oo,nA0 n=—oo,nA0 


Ss 


It is clear for this example that A, : H3(S') > H$~'(S'), for any s. Roughly the 
current is a derivative of potential and one degree of differentiability less smooth. 
Technically A, (for any positive y € C™°(2)) is a first order pseudo-differential 
operator [80]. The observation that for the example ee her = |n| + o(n-?) 
as |n| — oo for any p > —1 illustrates that the change in conductivity and 
radius of the interior object is of somewhat secondary importance! In the language 
of pseudodifferential operators for a general y such that y — 1 vanishes in a 
neighborhood of the boundary, A, and A, differ by a smoothing operator. 

Since (13), A is also a well-defined operator on L? —> L? with eigenvalues 
O(\n|~!) and is therefore a Hilbert-Schmidt operator. This is also known for the 
general case [36]. 

Early work on medical applications of EIT [58,74] hoped that the forward 
problem in EIT would be approximated by generalized ray transform — that is 
integrals along current stream lines. The example of a concentric anomaly was 
used to illustrate that EIT is nonlocal [99]. If one applies the voltage cos(@ + a), 
which for a homogeneous disk would result in current streamlines that are straight 
and parallel, a change in conductivity in a small radius p from the center changes 
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all measured currents, not just on lines passing through the region of changed 
conductivity |x| < p. In the 1980s a two-dimensional algorithm that backprojected 
filtered data along equipotential lines was popularized by Barber and Brown [11]. 
Berenstein [17] later showed that the linearized EIT problem in a unit disc can 
be interpreted as the Radon transform with respect to the Poincaré metric and a 
convolution operator and that Barber and Brown’s algorithm is an approximate 
inverse to this. 

In process applications of EIT and related techniques the term soft field imaging 
is used, which by analogy to soft field X-rays means a problem that is nonlinear 
and nonlocal. However in the literature when the “‘soft field effect” is invoked, it is 
often not clear if it is the nonlinear or nonlocal aspect to which they refer and, in the 
authors’ opinion, the term is best avoided. 


Measurements with Electrodes 


A typical electrical imaging system uses a system of conducting electrodes attached 
to the surface of the body under investigation. One can apply current or voltage 
to these electrodes and measure voltage or current respectively. For one particular 
measurement the voltages (with respect to some arbitrary reference) are V, and 
the currents 77, which are arranged in vectors as V and I ¢€ C¥. The discrete 
equivalent of the Dirichlet-to-Neumann A, map is the transfer admittance, or 
mutual admittance matrix Y which is defined by I= YV. 

It is easy to see that the vector 1 = (1,1,..., 1)" is in the null space of Y, 
and that the range of Y is orthogonal to the same vector. Let S be the subspace 
of C¥ perpendicular to 1; then it can be shown that Y|s is invertible from S to S. 
The generalized inverse (see chapter » Linear Inverse Problems) Z = Y* is called 
the transfer impedance. This follows from uniqueness of solution of shunt model 
boundary value problem. 

The transfer admittance, or equivalently transfer impedance, represents a com- 
plete set of data which can be collected from the L electrodes at a single frequency 
for a stationary linear medium. It can be seen from the weak formulation of (11) that 
Y and Z are symmetric (but for @ 4 0 not Hermittian). In electrical engineering this 
observation is called reciprocity. The dimension of the space of possible transfer 
admittance matrices is clearly no bigger than L(L — 1)/2, and so it is unrealistic 
to expect to recover more unknown parameters than this. In the analogous case 
of planar resistor networks with L “boundary” electrodes the possible transfer 
admittance matrices can be characterized completely [33], a characterization which 
is known at least partly to hold in the planar continuum case [63]. A typical electrical 
imaging system applies current or voltage patterns which form a basis of the space 
S, and measures some subset of the resulting voltages which as they are only defined 
up to an additive constant can be taken to be in S. 

The shunt model is nonphysical; in medical application with electrodes applied to 
skin and in “phantom” tanks used to test EIT systems with ionic solutions in contact 
with metal electrodes, a contact impedance layer exists between the solution or skin 
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and the electrode. This modifies the shunting effect so that the voltage under the 
electrode is no longer constant. The voltage on the electrode is still a constant Ve so 
now on £; there is a voltage drop across the contact impedance layer 


jpeg = Ve, (15) 
dv 

where the contact impedance zy could vary over E, but is usually assumed constant. 
Experimental studies have shown [56] that a contact impedance on each electrode 
is required for an accurate forward model. This new boundary condition together 
with (3) and (4) form the Complete Electrode Model or CEM. For experimental 
validation of this model see [30], theory [103], and numerical calculations [96, 111]. 
A nonzero contact impedance removes the singularity in the current density, 
although high current densities still occur at the edges of the electrodes (Fig. 4). 
For further details on the singularity in the current density, see [32]. While “point 
electrodes,” in which the current density is a sum of delta functions, are a limiting 
case of the CEM, they are not physically realistic as they result in nonphysical 
potentials not in H!(Q). The trace on the boundary cannot be evaluated at a point, 
so point measurements of voltage are undefined. However it can be shown that if 
the conductivity is changed only in the compliment of a neighborhood of dQ the 
resulting voltage difference at the boundary can be evaluated at points [52]. 

The set of imposed current patterns, or excitation patterns, is designed to span 
S, or at least that part of it that can be accurately measured in a given situation. In 
medical EIT, with process ERT following suit, early systems designed at Sheffield 
[11] assumed a two-dimensional circular domain. Identical electrodes were equally 
spaced on the circumference and, taking them to be numbered anticlockwise, the 
excitation patterns used were adjacent pairs, that is proportional to 


1 i= 
= )-1li=l+4+1 (16) 
0, otherwise , 


fori = 1,...L—1. The electronics behind this is balanced current source connected 
between two electrodes [59, Chap.2], and this is somewhat easier to achieve in 
practice than a variable current source at more than two electrodes. For general 
geometries, where the electrodes are not placed on a closed curve, other pairs of 
electrodes are chosen. For example [j = —1, while Ij = 4jc,£ # 1. 

Measurements of voltage can only be differential and so voltage measurements 
are taken between pairs of electrodes, for example adjacent pairs, or between each 
and some fixed electrode. In pair drive systems, similar to the original Sheffield 
system, voltages on electrodes with nonzero currents are not measured, resulting in 
incomplete knowledge of Z. 

In geophysical surface resistivity surveys it is common to use a pair drive and 
pair measurement system, using electrodes in a line where a two-dimensional 
approximation is used, or laid out in a rectangular or triangular grid where the 
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Fig. 4 The current density on the boundary with the CEM is greatest at the edge of the electrodes, 
even for passive electrodes. This effect is reduced as the contact impedance increases. Diagrams 
courtesy of Andrea Borsic. (a) Current density on the boundary for passive and active electrodes. In 
fact there is a jump discontinuity at the edge of electrodes for nonzeros contact impedance although 
the plotting routine has joined the /eft and right limits. (b) The effect of contact impedance on the 
potential beneath an electrode. The potential is continuous. (c) Interior current flux near an active 
electrode. (d) Interior current flux near a passive electrode 


full three-dimensional problem is solved. Measurements taken between pairs of 
non-current carrying electrodes. The choice of measurement strategy is limited 
by the physical work involved in laying out the cables and by the switching 
systems. Often electrodes will be distributed along one line and a two-dimensional 
approximate reconstruction is used as this gives adequate information for less 
cost. A wider spacing of the current electrodes is used where the features of 
interest is located at a greater depth below the ground. In another geophysical 
configuration, cross borehole tomography, electrodes are deployed down several 
vertical cylindrical holes in the ground, typically filled with water, and current 
passed between electrodes in the same or between different bore holes. Surface 
electrodes may be used together with those in the bore holes. In some systems the 
current is measured to account for a non-ideal current source. 

In capacitance tomography a basis of voltage patterns is applied and the choice 
V;} = 6;¢ is almost universal. The projection of these vectors to S (denoted as an 
“electrode-wise basis”) is convenient computationally as a current pattern. 
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Given a multiple drive system capable of driving an arbitrary vector of currents 
in S (in practice with in some limits on the maximum absolute current and on 
the maximum voltage) there remains a choice of excitation patterns. While exact 
measurements of ZI’ for I’ in any basis for S is clearly sufficient, the situation is 
more complicated with measurements of finite precision in the presence of noise. 
If a redundant set of currents is taken, the problem of estimating Z becomes 
one of multivariate linear regression. The choice of current patterns is then a 
design matrix. Another approach seeks the minimum set of current patterns that 
results in usable measurements. Applying each current pattern and taking a set of 
measurements take a finite time, during which the admittivity changes. Without 
more sophisticated statistical methods (such as Kalman filters [112]), there are 
diminishing returns in applying redundant current patterns. Suppose that the total 
power V* ZI is constrained (to keep the patient electrically safe) and the current best 
estimate of the admittivity gives a transfer admittance Zeaic, then it is reasonable to 
apply currents I such that (Z — Zcai-)I is above the threshold of voltages that can be 
accurately measured and modeled. One approach is to choose current patterns that 
are the right generalized singular vectors of Z — Zeaic with singular values bigger 
than an error threshold. The generalized singular values are with respect to the norm 
||I||z := ||ZI|| on S and are the extrema of the distinguishability defined as 


| |(Z _ Zealc) || | 


‘ 17 
[Mla oe 


for I ¢ S. These excitation patterns are called “optimal current patterns” [45] and 
can be calculated from an iterative procedure involving repeated measurement. For 
circular disk with rotationally symmetric admittivity and equally spaced identical 
electrodes, the singular vectors will be discrete samples of a Fourier basis and these 
trigonometric patterns are a common choice for multiple drive systems using a 
circular array of electrodes. 


2 Uniqueness and Stability of the Solution 


Uniqueness of solution is very important in inverse problems, although when talking 
to engineers it is often better to speak of sufficiency of data to avoid confusion. 
Interestingly it is generally true that results that show insufficiency of data, that one 
cannot recover an unknown function even if an infinite number of measurements 
of arbitrary precision are taken, have more impact in applied areas. While there 
are still unsolved problems in the uniqueness theory for the EIT inverse problem, 
there has been considerable progress over the last three decades and many important 
questions have been answered. While for an isotropic real conductivity y (with 
certain smoothness assumptions for dimensions n > 3), y is uniquely determined 
by the complete data A, (see [10,24, 107]), an anisotropic conductivity tensor is 
not uniquely determined by the boundary data, although some progress on what 
can be determined in this case has been made (see [3, 6, 43, 79]). Aside from 
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knowing what can and cannot be determined with ideal data, there are two important 
ways the theoretical work has a practical impact. Firstly in some cases the proof 
of uniqueness of solution suggests a reconstruction algorithm. As for the two- 
dimensional case (below) the most effective approach (the so-called d-method) to 
uniqueness theory has now been implemented as a fast, practical algorithm. The 
other is an understanding of the instability and conditional stability of the inverse 
problem. This helps us to determine what a priori information is helpful in reducing 
the sensitivity of the solution to errors in the data. 

In 1980 Calderén published a paper with the title “On a inverse boundary 
value problem” [26], where he addressed the problem of whether it is possible to 
determine the conductivity of a body by making current and voltage measurements 
at the boundary. It seems that Calderén thought of this problem when he was 
working as an engineer in Argentina for the Yacimientos Petroliféros Fiscales 
(YPF), but it was only decades later that he decided to publish his results. This 
short paper is considered the first mathematical formulation of the problem. For a 
reprinted version of this manuscript, refer to [27]. The authors wish to recall also 
the work due to Druskin (see [38-40]) which has been carried on independently 
from Calder6n’s approach and has been devoted to the study of the problem from a 
geophysical point of view. 


The Isotropic Case 


Calderon's Paper 

Calder6n considered a domain Q in R", n > 2, with Lipschitz boundary 0Q. He 
took y be a real bounded measurable function in Q with a positive lower bound. Let 
Q,, be the quadratic form (associated with A,) defined by 


0,(¢) = (¢, A, ¢) = [ y |Vu|? dx, (18) 


where u € H'(Q) solves the Dirichlet problem (2). Physically Q,(@) is the 
Ohmic power dissipated when the boundary voltage ¢ is applied. The bilinear form 
associated with Q,, is then obtained by using the polarization identity 


By. ¥) = =1 0,6 +) — 2,4) — 0,(19| 


NI NIe 
—— 


} ff (rive +P = viene yivvP) ax} 


yVu-Vvdx, (19) 


ll 
S—— 


Q 
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where Lyv = 0 in Q and vilag = v € H2(dQ). Clearly a complete knowledge of 
any of A,, Q, and B, are equivalent. Calder6n considered the “forward” map 


Q:y— Q, 


and proved that Q is bounded and analytic in the subset of L°(Q) consisting of 
functions y which are real and have a positive lower bound. He then investigated 
the injectivity of the map and in order to do so, he linearized the problem. He in 
fact proved the injectivity of the Fréchet derivative of Q at y = 1. Here, a few 
details of the linearization for a general y are given. Let u be the solution to (2) and 
U = u-+w satisfy L,43U = 0, with U|jq = ¢. The perturbation in potential 
satisfies w|aq = 0, by considering the Dirichlet data fixed and investigating how the 
Neumann data varies when y is perturbed to y + 6. This yields 


Lsu+ Lyw+ Lgw = 0. (20) 


Now let G : H7!(Q) — Hj(Q) be the Green’s operator that solves the 
equivalent of Poisson’s equation for L, with zero Dirichlet boundary conditions. 
That is for g € H~!(Q), LyGg = g and G(g)|aq = 0, the operator equation is 
given 


(1 + GLs)w = —GLsu. (21) 


An advantage of using the L™ norm is that it is clear ||L3|| > 0 in the H'(Q) > 
H'!(Q) operator norm as ||6||oo — 0. This means one can choose 6 small enough 
that ||GLs|| < 1 (in the operator norm on H'!(Q)) and this ensures that the term in 
the bracket in (21) is invertible and the operator series in 


w= (3-cxs) u (22) 
k=1 


is convergent. This proves that the map y +> u and hence Q is not just C® but 
analytic with (22) its Taylor series. Thus, the linearization of the map y +> A, is 


ou 


a + Ollldlleo): (23) 
Vv 


0 
Ay+5? => A, + Ya GLgu + 6 


A strength of this argument is that it gives the Fréchet derivative in these norms, 
rather than just the Gateaux derivative. It is easy to deduce that the Fréchet derivative 
of Q at y in the direction 6 is given by 


dQ(y)3($) = [ |Vul? dx. (24) 
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In many practical situations it is more common to fix the Neumann boundary 
conditions and measure the change in boundary voltage as the conductivity changes. 
Suppose Lyu = 0, L,43U = 0,w = U — u with 


ydu/dv = (y +4)dU/dv = g € HT'?(8Q) 


then a similar argument to the above shows 


| cae - | §|Vul2 dx + O(|I8II2,). (25) 
an dv Q 


The polarization identity is often applied to (25) giving 


a 
wy dx = -| 5Vu- Vv dx + O(\|6|[20). (26) 
aq | (OV Q 


where L,yv = 0. This is often used in practice with 


yo = x5 /IEil— x0, /1E jl @n 
which represents the difference in the characteristic functions of a pair of electrodes. 
In the case of the shunt model this makes the left-hand side of (25) equal to change 
that occurs in the difference between voltages on that pair of electrodes when the 
conductivity is perturbed. The formula (25) and its relatives are referred to as the 
Geselowitz Sensitivity Theorem in the bioengineering literature. With the CEM (25) 
still holds, but with wu and v satisfying (15) [97]. 

Returning to Calder6n’s argument: for y = 1 one has Lju = V7u. To prove the 
injectivity of dQ(1) one must show that if the integral appearing in (24) vanishes 
for all the harmonic functions in Q, then 6 = 0 in Q. Suppose the integral in (24) 
vanishes for all u € H'(Q) such that V7u = 0 in Q, then 


/ 6Vu-Vv =0, (28) 
Q 
whenever V2u = V7v = 0 in Q. For any z € R” consider a € R"” such that 
|a| = |z|, a-z = 0 and consider the harmonic functions 

u(x) = eT taa-x) 


v(x) _ gies merx) (29) 
which is equivalent to choosing 


u(x) = e*'?, v(x) = e*"?, 
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where p € C” with 
p-p=0. 


Here the real dot product on complex vectors is used p+ p := p’ p. With the choice 
made in (29), (28) leads to 


ariel? f ae” dx = 0, foreach  z, 


therefore 5(x) = 0, for all x € Q. Calder6én also observed that if the linear 
operator dQ(1) had a closed range, then one could have concluded that Q itself was 
injective in a sufficiently small neighborhood of y=constant. However conditions on 
the range of dQ(1), that would allow us to use the implicit function theorem, are 
either false or not known. Furthermore if the range was closed, one could have also 
concluded that the inverse of dQ(1) was a bounded linear operator by the open 
mapping theorem. Calderén concluded the paper by giving an approximation for 
the conductivity y if 


y=1+6 


and 6 is small enough in the L® norm, by making use of the same harmonic 
functions (29). Calder6én’s technique is based on the construction of low frequency 
oscillating solutions. Sylvester and Uhlmann proved in their fundamental paper 
[107] aresult of uniqueness using high frequencies oscillating solutions of L,u = 0. 
Their solutions are of type 


u(x, £,t) =e" f y-2(1 + w(x, &,0)), 


which behaves (for high frequencies €) in the same way as the solutions used 
by Calderén. These oscillating solutions have come to be known as complex 
geometrical optics (CGO) solutions. Before going into more details of the use of 
CGO solutions, an earlier result using a different approach is given. 


Uniqueness at the Boundary 

In 1984 Kohn and Vogelius [75] proved that boundary values, and derivatives at 
the boundary, of a smooth isotropic conductivity y could be determined from the 
knowledge of Q,,. Their result is given by the following theorem. 


Theorem 1. Let Q be a domain in R" (n > 2) with smooth boundary dQ. Suppose 


yi € C®%(Q), i = 1,2 is strictly positive and that there is a neighborhood B of 
some x* € 0Q so that 


0,,(f) = 0,(f), forall f, f €H2(8Q),  supp(f) c B. 
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Then 


glel : gla 
ax® yi(x") = Aya 


V2(x*), Va. 


Theorem | is a local result in the sense that one only need to know Q, in a open 
set of the boundary in order to determine the Taylor series of y on that open set. The 
global reformulation of this result given in terms of A, is given below. 


Theorem 2. Let y; € C®(Q), i = 1,2 be strictly positive. If Ay = Az, then 


gla glel 


ox = axe 


¥2, on OQ, Va. 


For a sketch of the proof of Theorem 2 see [110, Sketch of proof of Theorem 4.1, 
pp 6]. This result settled the identifiability question in the real-analytic category 
of conductivities. Kohn and Vogelius have extended this result to piecewise real- 
analytic (e.g., piecewise constant) conductivities in [76]. The proof of this result 
is based on [75] together with the Runge approximation theorem for solutions of 
Lyu = 0. 


CGO Solutions for the Schrédinger Equation 

In 1987 Sylvester and Uhlmann [106, 107] constructed in dimension n > 2 CGO 
solutions in the whole space for the Schrddinger equation with potential g. Before 
giving their result, the well-known relation between the conductivity equation and 
the Schrédinger equation will be derived. This relationship is also important in 
diffuse optical tomography (see chapter » Optical Imaging). 


Lemma 1. Let y € C?(Q) be strictly positive, yielding 


1 1 
y L(y?) = V-4, (30) 
where 
V(y2) 
q=-—T: 
y2 
Proof of Lemma 1. 
Lyu=yV*ut Vy-Vu (31) 


therefore 
Vy-Vu 
Y 


yoiLyu = y2Vu + 


Nie 
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: 1 
Consider for w = y2u 


which proves (30). oO 


The term q is usually called the potential of the Schrodinger equation, by analogy 
with the potential energy in quantum mechanics, this definition being somehow 
confusing given that in EIT w is the electric potential. The results in [106, 107] 
state the existence of CGO solutions for the Schrddinger equation with potential 
q bounded and compactly supported in R”. This result is as given in [110], which 
relies on the weighted L? space L3(R") = {f : f(I+ |x|7)°| f(x) |? dx}. For 5 < 0 


this norm controls the “growth at infinity.” The Sobolev spaces H. . (R") are formed 
in the standard way from Li(R") 


HE (R") = {f ¢ W*(R") | D*f € L3(R"), forall |a| < k}, 


where « is a multi-index, D® f denotes the ath weak derivative of f and W*(R") 
is the set of k times weakly differentiable functions on R”. 


Theorem 3. Let q € L®(R"), n > 2, with q(x) = 0 for |x| > R > O and 
—1 <6 <0. Then there exists €(5) and such that for every p € C" satisfying 


p-p=0 
and 


Wa + beP)Paliccoan +1 
lel 7 


there exists a unique solution to 


(V7 —q)u=0 (32) 
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of the form 


u(x, p) = e*"? (1+ q(x, p)), (33) 


with Wq(-,p) € L3(R"). Moreover Wvq(-.p) € H}(R") and for 0 < s < 2 there 
exists C = C(n, 8,6) > 0 such that 


Cc 
Ilvo(-. Plln2 S p= (34) 


Sketch of the proof of Theorem 3. Let u be a solution of (32) of type (33), then w, 
must satisfy 


(V? + 2p-V —q)Wy = 4. (35) 


The idea is that Eq.(35) can be solved for wy by constructing an inverse for 
V* + 2p-V) and solving the integral equation 
p & gral eq 


Wq = (V? +2p-V)! (g(1 + W)) (36) 


for w,. For more details about how to solve the above equation, refer to [110, 
Lemma 5.2] where it is shown that the integral equation (36) can only be solved 
in L5(R") for large |p|. Oo 


Other approaches for the construction of CGO solutions for the Schrédinger 
equation have been considered in [36, 49]. The reader may refer to [110] for more 
details about references on this topic and a more in-depth explanation about the 
constructions of this kind of solutions. 


Dirichlet-to-Neumann Map and Cauchy Data for the Schrédinger 
Equation 

If 0 is not a Dirichlet eigenvalue for the Schrédinger equation, then the Dirichlet-to- 
Neumann map associated with a potential g can be defined by 


~ du 
A =S — ; 
g(f ) qv lac 
where u solves the Dirichlet problem 


(V7—q)u=0 inQ 
ulan = f. 


V2y1/2 


As a consequence of Lemma |, for any g = yi? 
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me 0 1 1 
Aq(f) = rd Ulan 


ay2 y-2 

=(2 (y-2u) + y? yw) lan 
ov ov 
1 7) O(y~2 

= E =a rt +y? r i 


l/ _,d 1 = 
5 (ys) laa f+ y'lan Ary laa f). 


So the two Dirichlet-to-Neumann maps Ay and A, are related in the following way 
~ ly _,dy 1 1 
Rif) = 5 (y*)loa f + y2laa Ay(y Plea f). (7) 
for any f € H2 (dQ). For g € L®(dQ), the Cauchy data are defined as the set 
du 1 2 : 
Cy = j | Hlac. 9, lae Jue H(Q), (V'-gu=0 in Q. 
If 0 is not an eigenvalue of V* — q, then C;, is the graph given by 


Cy = \(f Ag(f)) € 1} (82) x H-#(9Q)} 


The result so far is very general and holds in any dimension n > 2. In the rest 
of the discussion on the uniqueness of Calderén’s problem, a distinction is made 
between the higher dimensional case n > 3 and the two-dimensional one. 


Global Uniqueness for n > 3 

Sylvester and Uhimann proved in [107] a result of global uniqueness for C?(Q) 
conductivities by solving in this way the identifiability question with the following 
result. Their result follows in dimension n > 3 from a more general one for the 
Schr6édinger equation, which is useful in its own right for other inverse problems. 


Theorem 4. Let qj €¢ L™(Q), i=1, 2. Assume Cy, = Cy), then qi = q2. 
Proof of Theorem 4. This result has been proved by constructing oscillatory solu- 


tions of (V* —q;) u; = 0 in R” with high frequencies. Beginning with the following 
equality 


[a — qz)uyu2 = 0 (38) 
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is true for any u; € H'(Q) solution to 
(V-—q)u=0 in Q, §=1,2. 


Equality (38) follows by 


7) 0 
[@-aum = f (Fount) as, 
Q aQ Ov ov 


which can be easily obtained by the divergence theorem. This extends g; on the 
whole R” by taking g; = 0 on R” \ Q and taking solutions of 


(V-—q))u,=0 in R’, i= 1,2 
of the form 
uw =e (14 Yy(x.p;)), 1 =1,2, (39) 


with |p;| large. This type of solutions are known as CGO solutions. p;, 1 = 1,2 is 


chosen of type 
gf he 
1 —__— 
Pl 5 5) 


m=-44i(S). (40) 


with n, k, / € R" and satisfying 


II 
| 


nk=k-k=n-1=0, 9 |nP = |kP +10), (41) 


the choices of 7, k, 1 having been made so that p;-p; = 0,i = 1,2. With these 
choices of p;,i = 1,2, 


= elx'k (1 + Wa + Wan + Va Var) 


and therefore 


G@ Wen if egy — 42) Un + ¥en + Van Venda (42) 
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By recalling that 


IlWqilln2@) S 
|p; | 


and letting |/| — oo one obtains g; = q2 (see [110, proof of Theorem 6.2, pp 10)). 
oO 


As a consequence of this result, the result [107] stated below is finally obtained. 


Theorem 5. Let y; € C?(Q), y; strictly positive, i=1, 2. Tf Ay, = Aj, then y, = 


y2 in Q. 


y2? 


Theorem 5 has been proved in [107] in a straightforward manner by constructing 
highly oscillatory solutions to L,u = 0 in . In this chapter, the line of [110] 
is followed in the exposition of such result as a consequence of the more general 
Theorem 4. Such a choice has been made because of the clearer exposition made in 
[110]. 

One can proceed by showing that Theorem 4 implies Theorem 5 for the sake 
of completeness. The reader can find it also in [110]. The argument used is the 
following. Let y; € C*(Q) be strictly positive and A,, = A,,. Then by [75], 


yilan = Y2lae, 


ov dy2 


>=— lan = z=—Ia0, 
ov ov 


therefore (37) implies Cz, = Cg, i.e., 1 = q2 =: q because of Theorem 4. Recall 
that 


_ Vey? 


re a 


which leads to 
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with 


Therefore it must be that 
y= 2 in Q, 


by uniqueness of the solution of the Cauchy problem. 

The identifiability question was then pushed forward to the case of y € C!:!(Q) 
with an affirmative answer by Nachman et al. in 1988 [90]. Nachman extended 
then this result to domains with C!:! boundaries (see [88]). The condition on the 
boundary was relaxed to 0Q Lipschitz by Alessandrini in 1990 in [3]; he proved 
uniqueness at the boundary and gave stability estimates for y ¢ W!?(Q), with 
Pp > n by making use of singular solutions with an isolated singularity at the 
center of a ball. This method enables one to construct solutions of Lyu = 0 on 
a ball behaving asymptotically like the singular solutions of the Laplace—Beltrami 
equation with separated variables. His results hold in dimension n > 2. Results of 
global uniqueness in the interior were also found in [3] among piecewise analytic 
perturbations of y, giving an extension of Kohn and Vogelius result in [76] to 
Lipschitz domains. 

Going back to the issue of global uniqueness, Brown [22] relaxed the regularity 
of the conductivity to 3 + e derivatives, which was followed by the uniqueness 


result of Paivarinta et al. [95] for wie conductivities. Their result is based on the 
construction of CGO solutions for conductivities y € W!°(R”) (n > 2). Recalling 
in what follows, their construction of the CGO followed by their uniqueness result. 


Theorem 6 ([95]). Let y € W!°(R*), y strictly positive and y = | outside a 
large ball. Let —1 < 6 < 0, then for |p| sufficiently large there is a unique solution 
of 

div(yVu) =0 in R* 


of the form 
nae? (y? + Vy (xp). (43) 


with Wy € L3(R’). Moreover, fy has the form 
=a 
V(x, p) = (wolx.p) — 77?) + ei(x.p), (4) 
where Wo, @, Satisfy 


lim |l@o(x,)—Y7? lla: = 0 (45) 


|p| 00 
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and 


lim ||ai(x, p)||,2 = 0. 
|p| 00 4 
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(46) 


Here, the idea behind the proof of the above Theorem 6 is recalled. The first step 


is to rewrite the conductivity equation 
div(yVu) =O in R" 
as 
Au+A-Vu=0 in R’, 
where 
A+Vlogy € L™(R") 


has compact support. By introducing 


A, = ®,* A 


@o(x,é) =e 2 
and with the above choice of wo one can show that 
: Ai 
lim || © — y~? Ili = 9. 
Let p € C” be such that p- p = O and define the operators 


Apu:=e * PA(e*' Pu) = Au+ 2p: Vu 
Vpui=e * PV(e*'Pu) = Vu pu. 


One can then define for any f € Cj°(R") 


_ 1 ix-¢ —f{@_ 
Ao'f = fe EP +2i0= dé, 


(47) 


(48) 
(49) 
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which can then be extended to a bounded operator 
Ay : As, (R") > A; (R"), 
for any —1 < 6 < Oands > 0. Moreover 


C(s,6,n) 


Ao las, +#¢ < ———, 
p b+1 5 |p| 


for some C > 0. The idea is now to construct w solution to 
(A, + A: V,)o1 = —(A,p + A: Vp)@0, (50) 


‘ (x) 
recalling that wo(x, €) = e7 “> and therefore depends on «. If one sets now 


52. K=1 8 
OQ, = As Qo, 


one can then rewrite (50) as 


(I+ A-V,A5')6= fi, (51) 
_—_—_—_—_ 
:=Tp(y) 


where 
fe t= —(Ap + A-V_)@o 


ge 


1 1 1 
= —e~7 (-—~Ag, + —(V@,)’ — ~A- Vo, + (A—A,)-p }. 
e (-Fa0. + 706.) go ee a) 
An approximate inverse of T,(y) is given by 


Sp(y) = y~2( — AV 85)? 
ail = 1 
= y ?T,(y')y?, 


therefore (51) has a unique solution in an appropriate space. To study now the 
behavior of w, as € + 0 and |p| > ov, recall that 


or (x,€,p) = Az'Spfe + Ap (T, | — Sp) fe (52) 
—— 
:=hp 
and now one can show that 


lim olla cery —_ 0, (53) 


|p| 00 
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which concludes the proof. Theorem 43 is then used in [95] to prove global 
uniqueness for conductivities y € Wie (Q), where Q is a bounded domain in 
R”, with n > 3. More precisely they prove the following theorem. 


Theorem 7. Letn > 3. Let y; € W2(Q) be strictly positive on Q, i = 1,2. If 
Ay, = Ay, 


then 


Vi=y2 on Q. 


Again, the idea is to give to the reader a flavor of how Theorem 7 is proven in 
[95]. The main idea is that if y; ¢ W'°(Q) and Ay, = Ay, witha; = JV, 
i = 1,2, then the identity 


i; (Va, -V(uyv) — Vaz: V(urv)) -| (V(aiu,)-Vv — V(azgu2)- Vv) dx = 0, 
Q Q 

(54) 
holds true Vv € H!(Q), Vu; € H'(Q) solution to div(y;Vu;) = 0 in Q = 1,2. 
The reader should notice that so far the results obtained hold true for conductivities 
of type y € W!©(Q), It is at this stage that one needs to assume 


y E W2?(R") 
to show the following technical results for @)(x, pe) = @ (x, &, pe) as in (52), 
€= |p|"! 
lim |e §Vy2-Va; dx = 0. (55) 


|p|0o 


With this choice of w,, by substituting the CGO solutions (44) into identity (54), 
one then can gain the desired uniqueness result. 

The above result was then followed by uniqueness for Wie (with p > 2n) 
in [23]. Recently Haberman and Tataru [48] proved that uniqueness holds for C! 
conductivities and Lipschitz conductivities close to the identity. Their result is the 
following. 


Theorem 8. Letn > 3 and Q C R" be a bounded domain with Lipschitz boundary. 
Let y; € W'°(Q) be a real conductivity, i = 1,2. Suppose there exists a constant 
C = C(n, Q) such that y;, i = 1,2 satisfies either 


I|V log villLoowa) SC (56) 
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or 
yi € C'1(Q). (57) 
If 
Ay, = Ay, 
then 
Yi = /2. 


Global Uniqueness in the Two-Dimensional Case 

The two-dimensional inverse conductivity problem must often be treated as a special 
case. Although results in [76] gave a positive answer to the identifiability question 
in the case of piecewise analytic conductivities, it was not until 1996 that Nachman 
[89] proved a global uniqueness result to Calder6n problem for conductivities in 
W??(Q), for some p > 1. An essential part of his argument is based on the 
construction of the CGO solutions and the d-method (sometimes written “d-bar 
method”) in inverse scattering introduced in one dimension by Beals and Coifman 
(see [14,15]). The result of [89] has been improved in 1997 for conductivities having 
one derivative in an appropriate sense (see [24]) and the question of uniqueness 
was settled in L°°(Q2) finally by Astala and Paivarinta [10] using d-methods. They 
proved 


Theorem 9. Let Q be a bounded domain in R? and y; € L~,i = 1,2 be real 
functions such that for some constant M, M~! < y; < M. Then 


Ay =Ay => vi = 2. 


Let us first explain the complex version of the problem used by [10]. Using the 
complex variable z = x, + ix2, and the notation d = 0/dz,d = 0/dz. Then the 
following result is available [10]: 


Lemma 2. Let Q be the unit disk in the plane and u € H'(Q) be a solution of 
Lyu = 0. Then there exists a real function v € H'(Q), unique up to a constant, 
such that f = u + iv satisfies the Beltrami equation 


af = of. (58) 


where uw = (1—y)/( + y). 
Conversely if f € H'(Q) satisfies (58), with a real valued 1, then u = Re f 
andv = Im f satisfy 


Lyu=0 and L,-1v=0, (59) 


where y = (1—u)/(1 + p). 
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Astala and Paivarinta reduce the general case of Q to that of the disk, and show that 
the generalized Hilbert transform H, : ulaq +> vlaq uniquely determines, and is 
determined by A,,. They go on to construct CGO solutions to (58) of the form 


fulz,k) = el®* (: +0 (<)) as |z| > 00 (60) 


and using a result connecting pseudoanalytic functions with quasi-regular maps 
prove that 1, determines jz. The original method Nachman used to prove unique- 
ness has resulted in the development of 4 reconstruction methods which are 
described below (section “Direct Nonlinear Solution”). See also the work of Druskin 
[38], which provides some answers to the 2-D geophysical settings. 


Some Open Problems for the Uniqueness 

One of the main open problems in dimension n > 3 is to investigate whether 
global uniqueness holds for the minimal assumption y € L®(Q2) or else to find 
what are the minimal assumptions on y in order to guarantee uniqueness. The 
inverse conductivity problem makes sense for conductivities that are indeed merely 
L©. There are neither proofs nor counter-examples for this in any dimension, to 
the authors’ knowledge, but it has been conjectured by Uhlmann that the optimal 
assumption is that the conductivities are Lipschitz. These open problems influence 
of course also the stability issue of finding appropriate assumptions (possibly on y) 
in order to improve the unstable nature of EIT. This issue will be studied in the next 
section. 


Stability of the Solution at the Boundary 
The result of uniqueness at the boundary of Theorem 2 has been improved in [108] 
to a stability estimate. The result is the following. 


Theorem 10. Let y; € C®(Q), i = 1,2, satisfy 


1 
eg ee i=1,2 (61) 
lIvillog@ SE, 1 = 1,2. (62) 
Given any0 <0 < arae there exists C = C(Q, E,n,o) such that 
Ilvi — y2||zeecaay S Cl|Ay, — Ayo (63) 
and 
dvi OY2 
—-— <C\|A,, —A,,|I%, 64 
Pe Felsen) = Cdn — Aol (64 
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where ||-||* denotes the norm in the space of bounded linear operators from 


H2(9Q) to H~2(aQ). 


This result improves the one of Theorem 2 in the sense that it is no longer 
necessary to require y € C™(Q) to determine y itself and its derivative at the 
boundary. It is only needed that y be continuous on Q to determine the boundary 
values of y, where if y € C!(Q) then one can determine y and its first derivative 
on 0Q as well. Subsequent results of stability at the boundary along the same lines 
have been proved in [3, 7,22, 43, 88, 91]. 


Global Stability for n > 3 


In 1988 Alessandrini [2] proved that, in dimension n > 3, under an a priori 
assumption on y of type 


H3(Q) < E, for some s > — +2, 
halen ; 


y depends continuously on A, with a modulus of continuity of logarithmic type. 
The result is stated below. 


Theorem 11. Let n > 3. Suppose that s > 5 and that yi € C(Q), i = 1,2isa 
conductivity satisfying 


0<tsnsk. i=1,2 (65) 

Ivillaste@ 52, i= 1,2. (66) 

Then there exists C = C(Q, E,n,s) andt = t(n,8), withO < t < 1 such that 
llvi — vallzcecay SC (|log ||Ay, — Aye |lal™ + [Ay — Ay) - (67) 


It has been proved [3, 4] that a similar stability estimate holds if (66) is replaced 
by 


Ilvillw2c0@) < E, b= 1,2. (68) 


Mandache [86] proved that logarithmic stability is optimal for dimension n > 2 
if the a priori assumption is of the form 


Il cea) SE; (69) 


for any finite k = 0,1,2,.... One of the main open problems in the stability issue 
is then to improve this logarithmic-type stability estimate under some additional a 
priori condition. In [8] it has been shown that (67) can be improved to a Lipschitz- 
type estimate in the case in which y is piecewise constant with jumps on a finite 
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number of domains. For piecewise constant complex conductivities a similar result 
has been proved in [16], where piecewise constant potentials of the Schrodinger 
equation have been investigated in [18] and Lipschitz stability estimates have been 
proved in this case as well. For more in-depth discussion about the stability in EIT 
and open problems in that regard refer to [5]. A similar estimate to (67) for the 
potential case can be found in [110]. 


Global Stability for the Two-Dimensional Case 

Logarithmic-type stability estimates in dimension n = 2 were obtained by [12, 13, 
82]. The results obtained in [82] last require only y be Hélder continuous of positive 
exponent 


Ilvlcea <= F, (70) 
forsomea,0 <a <1. 


Some Open Problems for the Stability 

The main open problem is to improve the logarithmic-type estimate found in [2] in 
any dimension n > 2. One approach would be to investigate whether the a priori 
regularity assumptions (68) can be further relaxed. On the other hand, since it has 
been observed [86] that this logarithmic type of estimate cannot be avoided under 
any a priori assumption of type (69) for any finite k = 0,1,2,..., it seems natural 
to think that another direction to proceed would be the one of looking for different a 
priori assumptions rather than the one of type (69). For a complete analysis of open 
problems in this area, refer to [5]. 


The Anisotropic Case 


The Non-uniqueness 

In anisotropic media the conductivity depends on the direction, therefore it is 
represented by a matrix y = (yj;)}, ;=,, Which is symmetric and positive definite. 
Anisotropic conductivity appears in nature, for example as a homogenization limit 
in layered or fibrous structures such as rock stratum or muscle, as a result of 
crystalline structure or of deformation of an isotropic material. Let Q C R” bea 
domain with smooth boundary dQ (a Lipschitz boundary will be enough in most 
cases). The Dirichlet problem associated in the anisotropic case takes the form 


~ Go #4) =0in 2 
j=] 


ij= 


Ulan = f, 


(71) 


where f € H 2(9Q) is a prescribed potential at the boundary. The Dirichlet-to- 
Neumann map associated with y is defined by 
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Ay f = yVu-viag, (72) 
for any u solution to (71). Here yVu-v = at (Vij a) v;|aq and as usual v = 


(v;)_, is the unit outer normal to dQ. The weak formulation of (72) is commonly 
used and will be given below for the sake of completeness. 


Definition 1. The Dirichlet-to-Neumann map associated with (71) is 
Ay : H2(92) —> H7~2(9Q) 


given by 
(Ay fin) = [ovovues) Yoon de (73) 


for any f, € H2(9Q), u,d € H'(Q), dlaa = 7 and u is the weak solution 
to (71). 


A conductor is isotropic when y = (y;;) is rotation invariant, i.e., when at each 
point 


R'yR =y, 


for all rotations R. This is the case exactly when y = a J, where a > 0 is a scalar 
function and / the identity matrix. 

In the section “The Isotropic Case,” the uniqueness problem for the isotropic case 
was considered solved; on the other hand, in the anisotropic case, A, does not in 
general determine y. Tartar (see [75]) observed the following non-uniqueness result. 


Proposition 1. If ~ : Q —> Q is a C! diffeomorphism such that w(x) = x, 
for each x € 0, then y and y = (Pe Dy ow! have the same Dirichlet-to- 
Neumann map. 


The proof of this result is given below as a tutorial for the first-time reader of this 
material. 


Proof. Let us consider the change of variables y = y(x) on the Dirichlet integral 


du du - du Ou 
[ic Oxf Ox O =f v0 ay! ae (74) 


where 
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(Dy) yD)" 


-1 
det(DW) oy (y) 


7) = 
and 
i(y) =uow'(y). 
Notice that the solution u of the Dirichlet problem 


V-yVu=O0in Q 
Ulan = f 

minimizes the integral appearing on the left-hand side of (74), therefore i = uoy ! 
minimizes the Dirichlet integral appearing on the right-hand side of the same. One 


can then conclude that u solves 


ee in Q 
lpg = f =uoy!. 


Let us consider now the solution v of 


oo Q 
vlan = 8 


and let v be obtained by v by the change of variable, therefore v solves 


ao in Q 
Vaga=Z=goyl. 


By the change of variables in the Dirichlet integrals, 


/ - du ov ax=f fy Ou vo 
go) ae og Je dy! dys "i 
which can be written as 
/ yVu-Wwdx = f yVu-Vvdy, 
Q Q 
which is equivalent to 
/ V-(vyVuy dx f WV(Vu)dx = [ V-7Vi) dy — f vV -(yVu) dy 
Q Q Q Q 


and by the divergence theorem 
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[ ovvwvas= f vy Vu-vds, 
aa aa 


butp =vow!=v=gandi=uoy! =u= f atthe boundary 0Q, then 


i gh,(f)ds = / gA;(f) ds 
dQ dQ 
then Ay — A;. oO 


Since Tartar’s observation has been made, different lines of research have been 
pursued. One direction was to prove the uniqueness of y up to diffeomorphisms that 
fix the boundary, whereas the other direction was to study conductivities with some 
a priori information. The first direction of research is summarized in what follows. 


Uniqueness up to Diffeomorphism 
The question here is to investigate whether Tartar’s observation is the only obstruc- 
tion to unique identifiability of the conductivity. A first observation is that the 
physical problem of determining the conductivity of a body is closely related to 
the geometrical problem of determining a Riemannian metric from its Dirichlet-to- 
Neumann map for harmonics functions [80]. 

Let (M, g) be a compact Riemannian manifold with boundary. The Laplace— 
Beltrami operator associated with the metric g is given in local coordinates by 


"| 


. _1 0 1 ;, Ou 
Ag i= > (det g) pede vat 


ij=l 


The Dirichlet-to-Neumann map associated with g is the operator A, mapping 
functions u|ay € H'/?(aM) into (n — 1)-forms A, (ula) € H7'/? (Q""!(aM)) 


Ag(f) = i*(#gdu), (75) 
for any u € H'(M) solution to Agu = 0 in M, with ulay = /f. Here i is the 


inclusion map i : dM — M and i* denotes the pull-back of forms under the map 
i. In any local coordinates (75) becomes 


n det 
LAS = » vi gt a Veet glow. (76) 
ij=l J 


The inverse problem is to recover g from Ag. In dimension n > 3, the 
conductivity y uniquely determines a Riemannian metric g such that 


VY = *g, (77) 
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where *, is the Hodge operator associated the metric g mapping |-forms on M into 
(n — 1)-forms (see [42, 80, 81]). In any local coordinates (77) becomes 


(gij) = det yer)™=2 (viz) and (yj) = det ger)? (¢"), (78) 


where (g“/), (y'/) denotes the matrix inverse of (g;;) and (y;;) respectively. It has 
been shown in [80] that if 7 is a domain in R”, then for n > 3 


Ay = Ay. (79) 
In dimension n > 3 if y is a diffeomorphism of M that fixes the boundary, 
Ayre = Ag, (80) 


where w* g is the pull-back of g under y. For the case n = 2 the situation is 
different as the two-dimensional conductivity determines a conformal structure of 
metrics under scalar field, i.e., there exists a metric g such that y = 9*,, fora 
positive function g. Therefore inn = 2, if y is a diffeomorphism of M that fixes 
the boundary, 


Agw*g = Ag, (81) 


for any smooth positive function such that g|ay = 1. It seems natural to think 
that (80) and (81) are the only obstructions to uniqueness forn > 3 andn = 
2 respectively. In 1989 Lee and Uhlmann [80] formulated the following two 
conjectures. 


Conjecture I. Let M bea smooth, compact n-manifold, with boundary, n > 3 and 
let g, g be smooth Riemannian metrics on M such that 


Ag=A 


gz 


Then there exists a diffeomorphism yw : M — M with Wlay = Id, such that 
saws. 


Conjecture 2. Let M be a smooth, compact 2-manifold with boundary, and let g, g 
be smooth Riemannian metrics on M such that 


Then there exists a diffeomorphism yw : M > M with wlay = I d, such that y* g 
is a conformal multiple of g, in other words there exists ¢ € C™(M) such that 


W g=$¢s. 
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Conjecture | has been proved in [80] in a particular case. The result is the 
following. 


Theorem 12. Let M be a compact, connected, real-analytic n-manifold with con- 
nected real-analytic boundary, and assume that 1 (M, 3M) = 0 (this assumption 
means that every closed path in M with base point in 0M is homotopic to some path 
that lies entirely in 0M ). Let g and & be real-analytic metrics on M such that 


and assume that one of the following conditions holds: 


1. M is strongly convex with respect to both g and &; 
2. either g or g extends to a complete real-analytic metric on a non-compact real- 
analytic manifold M (without boundary) containing M. 


Then there exists a real-analytic diffeomorphism vw : M > M with Wlay = Id, 
such that g = * g. 


Theorem 12 has been proved by showing that one can recover the full Taylor 
series of the metric at the boundary from Ag. The diffeomorphism y is then 
constructed by analytic continuation from the boundary. As previously mentioned, 
the full Taylor series of y was recovered by Kohn and Vogelius in [75] from the 
knowledge of A,, in the isotropic case and then a new proof was given in [106] by 
showing that the full symbol of the pseudodifferential operator A, determines the 
full Taylor series of y at the boundary. In [80] a simpler method suggested by R. 
Melrose consisting of factorizing Ag, is used. In 1990 Sylvester proved in [105] 
Conjecture 2 in a particular case. His result is the following. 


Theorem 13. Let 2 be a bounded domain in R? with a C? boundary and let y,, y2 
be anisotropic C? conductivities in Q such that 


|| log (det y;) |les< e(M, Q), for i =1, 2, (82) 
with M >|| y; \|c3, for i=1, 2 and e(M, &) sufficiently small. If 
Ay, = Ay, 
then there exists a C? diffeomorphism W of Q such that w\aq = Id and such that 
VeVi = 2 
Nachman [89] extended this result in 1995 by proving the same theorem but 


removing the hypothesis (82). In 1999 Lassas and Uhlmann [78] extended the result 
of [80]. They assumed that the Dirichlet-to-Neumann map is measured only on 
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a part of the boundary which is assumed to be real-analytic in the case n > 3 
and C-smooth in the two-dimensional case. The metric is here recovered (up to 
diffeomorphism) and the manifold is reconstructed. Since a manifold is a collection 
of coordinate patches, the idea is to construct a representative of an equivalent class 
of the set of isometric Riemannian manifolds (M, g). Recalling that if is an open 
subset of 0M, and defining 


Agr (f) = Ag(/lr. 
for any f with supp f C I’. The main result of [78] is given below. 
Theorem 14. Let us assume that one of the following conditions is satisfied: 


1. Mis a connected Riemannian surface; 
2. n => 3 and (M, g) is a connected real-analytic Riemannian manifold and the 
boundary 0M is real-analytic in the nonempty set T C 0M. 


Then 


1. For dim M = 2 the Ag p-mapping and T determine the conformal class of the 
Riemannian manifold (M, g). 

2. Fora real-analytic Riemannian manifold (M, g), dim M > 2 which boundary is 
real analytic inY, the Ag, p-mapping and I determine the Riemannian manifold 
(M, g). 


This result improved the one in [80] also because here the only assumption on 
the topology of the manifold is the connectedness, while in [80] the manifold was 
simply connected and the boundary of the manifold was assumed to be geodesically 
convex. Theorem 14 has been extended in [79] to a completeness hypothesis on M. 


Anisotropy Which Is Partially A Priori Known 

Another approach to the anisotropic problem is to assume that the conductivity y 
is a priori known to depend on a restricted number of unknown spatially dependent 
parameters. In 1984 Kohn and Vogelius (see [75]) considered the case where the 
conductivity matrix y = (y;;) is completely known with the exception of one 
eigenvalue. The main result is the following. 


Theorem 15. Let y, y be two symmetric, positive definite matrices with entries 
in L° (82), and let {yi}, {yi} and {e;}, {éi} be the corresponding eigenvalues and 
eigenvectors. For xo € 0Q, let B be aneighborhood of xo relative to 2, and suppose 
that 


vy ye C™(B); (83) 
dQ B is C™; (84) 
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ej =, A;=A; in B, for 1<j<n-1; (85) 
€n(Xo9)- V(X) 4 0. (86) 
If 
0,(¢) = Q;(¢) forevery ~ € H?(IQ), 
with supp C BM AQ, then 
D*Ay(X0) = Dk An (x0). 


for every k = (ky,....kn), ki € Zt, i =1,...,n. 


In 1990 Alessandrini [3] considered the case in which y is a priori known to be of 
type 


y(x) = A(a(x)), 


where t — A(t) is a given matrix-valued function and a = a(x) is an unknown 
scalar function. He proved results of uniqueness and stability at the boundary 
and then uniqueness in the interior among the class of piecewise real-analytic 
perturbations of the parameter a(x). The main hypothesis he used is the so-called 
monotonicity assumption 


D,A(t) > C I, 


where C > 0 is a constant. In 1997 Lionheart [81] proved that the parameter a(x) 
can be uniquely recovered for a conductivity y of type 


y(x) = a(x) Ao(*), 


where Ao(x) is given. Results in [3] have been extended in 2001 by Alessandrini 
and Gaburro [6] to a class of conductivities 


y(x) = A(x, a(x)), 


where A(x, f) is given and satisfies the monotonicity condition with respect to the 
parameter ¢ 


D,A(x, th> CTI, 
where C > 0 is a constant (see [6] or [41] for this argument). In [6] the authors 


improved results of [3] since they relaxed the hypothesis on A(-, ¢) for the global 
uniqueness in the interior and the result there obtained can be applied to [81] as well. 
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The technique of [6] can also be applied to the so-called one-eigenvalue problem 
introduced in [75]. Results of [6] have been recently extended to manifolds [43] 
and to the case when the local Dirichlet-to-Neumann map is prescribed on an open 
portion of the boundary [7]. 


Some Remarks on the Dirichlet-to-Neumann Map 


EIT with Partial Data 
In many applications of EIT one can actually only take measurements of voltages 
and currents on some portion of the boundary. In such situation the Dirichlet-to- 
Neumann map can only be defined locally. 

Let Q C R" be a domain with conductivity y. If is a nonempty open portion 
of 0Q, the subspace of H 2 (dQ) is introduced 


HiT) ={fe H2(3Q) | supp f CT}. (87) 


Definition 2. The local Dirichlet-to-Neumann map associated with y and IT is the 
operator 


AD: HAD) > (HE) ey) 


defined by 
(A, hn) = / yVu-V¢ dx, (89) 
Q 


ill 
for any f,n € H2(L), where u € H'(Q) is the weak solution to 


V-(yv(x)Vu(x)) = 0, in Q, 
u=f, on dQ, 


and @ € H'!(Q) is any function such that ¢|3q = 7 in the trace sense. 


Note that, by (89), it is easily verified that A is self adjoint. The inverse problem 
is to recover y from a: 

The procedure of reconstructing the conductivity by local measurements has 
been studied first by Brown [22], where the author gives a formula for reconstructing 
the isotropic conductivity pointwise at the boundary of a Lipschitz domain Q 
without any a priori smoothness assumption of the conductivity. Nakamura and 
Tanuma [91] give a formula for the pointwise reconstruction of a conductivity 
continuous at one point x° of the boundary from the local D-N map when the 
boundary is C! near x°. Under some additional regularity hypothesis the authors 
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give a reconstruction formula for the normal derivatives of y on dQ at x° € dQ 
up to a certain order. A direct method for reconstructing the normal derivative of 
the conductivity from the local Dirichlet-to-Neumann (D-N) map is presented in 
[92]. The result in [91] has been improved by Kang and Yun [70] to an inductive 
reconstruction method by using only the value of y at x°. The authors derive 
here also Holder stability estimates for the inverse problem to identify Riemannian 
metrics (up to isometry) on the boundary via the local D-N map. An overview on 
reconstructing formulas of the conductivity and its normal derivative can be found 
in [93]. 

For related uniqueness results in the case of local boundary data, refer to 
Alessandrini and Gaburro [7], Bukhgeim and Uhlmann [25], Kenig et al. [73] and 
Isakov [68], and, for stability, [7] and Heck and Wang [55]. Recent results are 
also provided by Kenig and Salo [71,72]. It is worth noting that [72] generalizes 
the results obtained in both [73] and [68] by making use of improved Carleman 
estimates with boundary terms, CGO solutions involving reflected Gaussian beam 
quasimodes and invertibility of (broken) geodesics ray transforms. Results of 
stability for cases of piecewise constant conductivities and local boundary maps 
have also been obtained by Alessandrini and Vessella [8], and by Di Cristo [35]. 
Another useful reference is [110, Sect. 7]. 


The Neumann-to-Dirichlet Map 

In many applications of EIT especially in medical imaging, rather than the local 
Dirichlet-to-Neumann map, one should consider the so-called local Neumann-to- 
Dirichlet (N-D) map. That is, the map associating the specified current densities 
supported on a portion of T C dQ to the corresponding boundary voltages, also 
measured on the same portion I of dQ. Usually electrodes are only applied to part 
of the body. Geophysics, of course, gives an extreme example where I is a small 
portion of the surface of the earth Q. It seems appropriate at this stage to recall the 
definition of the N-D map and its local version for the sake of completeness [7]. 

Let us introduce the following function spaces (see [7]) 


#3 (02) =} HA(a0) [9 =o 
Hz * (02) = {y € H74(@Q)| (y, 1) = 0}. 


Observe that, if considering the (global) D-N map A,, that is the map introduced 
in (88) A in the special case when T = 0Q, then it maps onto oH~2(9Q), and, 


1 
when restricted to Hj (dQ), it is injective with bounded inverse. Then one can define 
the global Neumann-to-Dirichlet map as follows. 


_1 
Definition 3. The Neumann-to-Dirichlet map associated with y, Ny, : H, ? (0&2) 


1 
— H (dQ) is given by 
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aS (viene a ; (90) 


Note that N, can also be characterized as the self-adjoint operator satisfying 
ie NW) = fF yoy VuGx)- Vue dx, 01) 
Q 


a 
for every Wy € H, ?(0Q), where u € H'!(Q) is the weak solution to the Neumann 
problem 


L,u= 0, in Q, 
yVu-vilag = ¥, on OQ, (92) 
Sig u = 9. 


At this point, it is useful to introduce the local version of the N-D map. Let T° be 
_ 1 
an open portion of dQ and let A = dQ \ I’. Here, Hy) (A) denotes the closure in 
1 
H2 (OQ) of the space H(A) previously defined in (87) and introducing 


H>?(P) =/Jwe H>?(0Q)| (w, f) =0, forany f € H3(A) : (93) 


that is the space of distributions w € H 3 (9Q) which are supported in I and have 
zero average on dQ. The local N-D map is then defined as follows. 


Definition 4. The local Neumann-to-Dirichlet map associated with y, I’ is the 
operator Ny : oH-2(T) > (4-2 (r))" oH 2(8Q) given by 


(Ny i, J) = Wy i. J), (94) 


for every i,j € oH~2(F). 


3 The Reconstruction Problem 
Locating Objects and Boundaries 


The simplest form of the inverse problem is to locate a single object with a 
conductivity contrast in a homogeneous medium. Some real situations approximate 
this, such as a weakly electric fish locating a single prey or the location of an 
insulating land mine in homogeneous soil. Typically the first test done on an EIT 
system experimentally is to locate a cylindrical or spherical object in a cylindrical 
tank. Linearization about y = | simplifies to 


742 A. Adler et al. 


V-w = -V8-Vut O (|I5||Z00) - (95) 


Here, the disturbance in the potential w is, to first order, the solution of Poisson’s 
equation with a dipole source centered on the object oriented in the direction 
of the unperturbed electric field. With practice experienced experimenters (like 
electric fish) can roughly locate the object from looking at a display of the voltage 
changes. When it is known a priori that there is a single object, either small or 
with a known shape, the reconstruction problem is simply fitting a small number of 
model parameters (e.g., position, diameter, conductivity contrast) to the measured 
voltage data, using an analytical or numerical forward solver. This can be achieved 
using standard nonlinear optimization methods. For the two-dimensional case a fast 
mathematically rigorous method of locating an object (not required to be circular) 
from one set of Cauchy data is presented by Hanke [50]. Results on the recovery 
of the support of the difference between two piecewise analytic conductivities by 
solving the linearized problem are given in [53]. 

In the limiting case where the object is insulating, object location becomes a free 
boundary problem, where the Dirichlet-to-Neumann map is known on the known 
boundary and only zero Neumann data known on the unknown boundary. This 
is treated theoretically for example by [61] and numerically in [62]. A practical 
example is the location of the air core of a hydrocyclone, a device used in chemical 
engineering [114]. 

In more complex cases the conductivity may be piecewise constant with a jump 
discontinuity on a smooth surface. In that case there are several methods that have 
been tested at least on laboratory data for locating the surface of the discontinuity. 
One would expect in general that the location of a surface can be achieved more 
accurately or with less data than the recovery of a spatially varying function and 
this is confirmed by numerical studies. 

A natural method of representing the surface of discontinuity is as a level set 
of a smooth surface (see chapter » Level Set Methods for Structural Inversion 
and Image Reconstruction). This approach has the advantage that no change in 
parameterization is required as the number of connected components changes, in 
contrast for example to representing a number of star-shaped objects using spherical 
polar coordinates. The approach to using the level set method in EIT is exactly the 
same as its use in scattering problems apart from the forward problem (see chapter 

Inverse Scattering). Level set methods have been tested on experimental ERT and 
ECT data by Soleimani et al. [102]. Here some of their results are reproduced in 
Fig. 5. 

Another approach to locating a discontinuity, common with other inverse 
boundary value problems for PDEs are “sampling and probe methods” in which 
a test is performed at each point in a grid to determine if that point is in 
the object. Linear sampling and factorization methods are treated in the chapter 

Sampling Methods. Theory and numerical results for the application of Linear 
Sampling to ERT for a half space are given by Hanke and Schappel [51]. Sampling 
methods generally require the complete transfer impedance matrix and where only 
incomplete measurements are available they must be interpolated. 
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(a) (b) (c) 


(a) Level set reconstruction (c) from experimental ERT data for high contrast objects (a) compared 
with generalized Tikhonov regularization. 


(a) (b) (c) 


(b) Level set reconstruction (c) from experimental ECT data for a pipe (a) compared with gener- 
alized Tikhonov regularization. 


Fig. 5 Comparison of level set reconstruction of 2D experimental data compared to generalized 
Tikhonov regularization using a Laplacian smoothing matrix (EIDORS-2D [113]). Thanks to 
Manuchehr Soleimani for reconstruction results, and Wu Quaing Yang and colleagues for the ECT 
data [116], the experimental ERT data was from [113] 


Also in the spirit of probe methods is the monotonicity method of Tamburrino 
and Rubinacci [109]. This method follows from the observation that for y real the 
map y +> Z, is monotone in the sense that y; < y2 > Z,, — Z,, = 0, where a 
matrix Z > 0 if its eigenvalues are non-negative. Suppose that for some partition 
{Q;} of Q (e.g., pixels or voxels) 


v=o vixe, (96) 


and each y; € {m, M},0 < m < M. Foreachi let Z’” be the transfer impedance for 
a conductivity that is M on Q; and m elsewhere. Supposing Z — Z/" has a negative 
eigenvalue, then y; = m. For each set in the partition the test is repeated, and it 
is inferred that some of the conductivity values are definitely m. The equivalent 
procedure is repeated for each Z . In practice, for large enough sets in the partition 
and M —m big enough, most conductivity values in the binary image are determined, 
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although this is not guaranteed. In practice the method is very fast as Zi” and vig can 
be precomputed and one only needs to find the smallest eigenvalue of two modestly 
sized matrices for each set in the partition. In the presence of noise, of course, one 
needs a sufficiently negative eigenvalue to be sure of the result of the test, and the 
method does assume that the conductivity is of the given form (96). Recently a 
partial converse of the monotonicity result has been found [54] and this promises 
more accurate fast methods of reconstructing the shape of an inclusion. 

If conductivities on some sets are undetermined, they can then be found using 
other methods. For example [115] use a Markov Chain Monte Carlo method to 
determine the expected value and variance of undetermined pixels in ECT. 


Forward Solution 


Most reconstruction algorithms for EIT necessitate solution of the forward problem, 
that is to predict the boundary data given the conductivity. In addition, methods 
that use linearization typically require electric fields in the interior. The simplest 
case is an algorithm that uses a linear approximation calculated at a homogenous 
background conductivity. For simple geometries this might be done using an 
analytical method, while for arbitrary boundaries Boundary Element Method is a 
good choice, and is also suitable for the case where the conductivity is piecewise 
constant with discontinuities on smooth surfaces. For general conductivities the 
choice is between finite difference, finite volume and finite element methods 
(FEMs). All have been used in EIT problems. FEM has the advantage that the 
mesh can be adapted to a general boundary surface and to the shape and location 
of electrodes, whereas regular grids in finite difference/volume methods can result 
in more efficient computation, traded off against the fine discretization needed to 
represent irregular boundaries. One could also use a hybrid method such as finite 
element on a bounded domain of variable conductivity coupled to BEM for a 
homogeneous (possibly unbounded) domain. 

In reconstruction methods that iteratively adjust the conductivity and resolve the 
forward problem, a fast forward solution is needed, whereas in methods using a 
linear approximation, the forward solution can be solved off-line and speed is much 
less important. 

The simplest, and currently in EIT the most widely used, FE method is first order 
tetrahedral elements. Here a polyhedral approximation Q) to Q is partitioned into 
a finite set of tetrahedra 7;,, k = 1,...,m; which overlap at most in a shared face, 
and with vertices x;,i = 1 < n,. The potential is approximated as a sum 


un(x) = D> wigi (x), (97) 


where the ¢; are piecewise linear continuous functions with @; (x; ) = 4;;. The finite 
element system matrix K € C”’*"" is given by 


Electrical Impedance Tomography 745 


Kij = i; yV di Vo; dx. (98) 
Qp 

On each tetrahedron V¢; is constant which reduces calculation of (98) in the 

isotropic case to the mean of y on each tetrahedron. One then chooses an 

approximation to the conductivity in some space spanned by basis functions yj; (x) 


v= ovivi. (99) 


One can choose these functions to implement some a priori constraints such as 
smoothness and to reduce the number of unknowns in the discrete inverse problem. 
Or one can choose basis functions just as the characteristic functions of the 
tetrahedra, which makes the calculation, and updating, of the system matrix very 
simple. In this case, all a priori information must be incorporated later, such as by a 
regularization term. In general the integrals 


/ Wi Voi - Vo; dx (100) 
Qp 


are evaluated using quadrature if they cannot be done explicitly. If the inverse 
solution uses repeated forward solutions with updated conductivity but with a fixed 
mesh, the coefficients (100) can be calculated once for each mesh and stored. For a 
boundary current density 7 = yVu-v the current vector Q € R”’ is defined by 


qi = [ia dx (101) 
IQ 
and the FE system is 
Ku = Q, (102) 


where u is the vector of u;. One additional condition is required for a unique 
solution, as the voltage is only determined up to an additive constant. One way 
to do this is to choose one (“grounded”) vertex ig and enforce u;, = 0 by deleting 
the i, row and column from the system (102). It is clear from (98) that for a pair 
of vertices indexed by i,j that are not both in any tetrahedron, K;; = 0. The 
system (102) is equivalent to Ohm’s and Kirchoff’s law for a resistor network with 
resistors connecting nodes i and 7 when the corresponding vertices in the FE mesh 
share and edge (where some dihedral angles are obtuse, there is the possibility of 
negative conductances). It is worth noting that whatever basis is user to represent the 
approximate conductivity (including an anisotropic conductivity) the finite element 
system has only one degree of freedom per edge and one cannot hope, even with 
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perfect data and arithmetic, to recover more than 1, (the number of edges) unknowns 
from the discretization of the inverse problem. 

The above formulation implements the shunt model. The CEM with specified 
currents can be implemented following Vauhkonen [111] using an augmented 
matrix. Defining 


L 
I 
Ki, = Ky +) -{ iP j dX, 103 
ij ij Lz, oe (103) 


where, here, | E;| denotes the area of the /th electrode, and 


: 1 
Ki, — —|E¢| for £ => 1,...,L, (104) 
Xe 
0d 1 ; 
Ko? = - —; dx i=1,..,.n,€=1,..,L. (105) 
Ejl <£ 


The system matrix for the CEM, KE € Ct) x(t) jg 


a K° K° 
Ko™M — re Koy (106) 


In this notation, the linear system of equations has the form 


K@Mg = Q, (107) 


where U = (u1,...,Un,,Vi, Vz)? and Q = (0,...,0,4),...,1,)". The constraint 
V e€ S (see section “Measurements with Electrodes”) is often used to ensure 
uniqueness of solution. The transfer impedance matrix is obtained directly as 


although it is usual to solve the system (106) as wu in the interior is used in the 
calculation of the linearization. This formulation should only be used for reasonably 
large ze, as small ze will result in the block K a dominating the matrix. For an 
accurate forward model it is necessary to estimate the contact impedance accurately. 
This is more important when measurements from current carrying electrodes are 
used in the reconstruction, or when the electrodes are large (even if they are 
“passive” Ig = OQ). The CEM boundary condition is rather unusual and most 
commercial FE systems will not include the boundary condition easily. This is 
one of the reasons forward solution code for EIT is generally written specifically 
for the purpose, such as the EIDORS project [1]. It is possible to calculate the 
transadmitance matrix Y = Z* more easily with standard solvers. One sets Robin 
boundary u + zeydu/dv = V, on each Ey and the zero Neumann condition (4) 
using V forming a basis for S, one then takes the integral of the current over each 
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electrode as the current I = YV. For a given current pattern I one applies the Robin 
conditions V = Y‘I and the solver gives the correct u. Advantages of commercial 
solver are that they might contain a wide variety of element types, fast solvers, 
and mesh generators. Disadvantages are that they may be hard to integrate as part 
of a nonlinear inverse solver, and it might be harder to calculate the linearization 
efficiently. 

In fact implementing code to assemble a system matrix is quite straightforward; 
much harder for EIT is the generation of three-dimensional meshes. For human 
bodies with irregular boundaries of inaccurately known shape this is a major 
problem. To apply boundary conditions accurately without overfine meshes it is also 
important that the electrodes are approximated by unions of faces of the elements. 
While the accuracy of the FEM is well understood in terms of the error in the 
solution u, in EIT it is necessary to require that the dependence of the boundary 
data on the conductivity be accurate, something that is not so well understood. In 
addition if the conductivities vary widely it may be necessary to remesh to obtain 
the required accuracy, and ideally this capability with be integrated with the inverse 
solver [87]. 


Regularized Linear Methods 


Methods based on linearization are popular in medical and process versions of 
EIT. The reasons are twofold. Process and medical applications benefit from very 
rapid data acquisition times with even early systems capable measuring a transfer 
impedance matrix in less that 0.04 s, and it was often required to produce an image in 
real time. The application of a precomputed (regularized) inverse of the linearized 
forward problem required only about $L*(L — 1)* floating point operations. For 
reasons of both speed and economy early systems also assumed a two-dimensional 
object with a single ring of electrodes arranged in a plane. The second reason 
for using a linear approximation is that in medical applications especially there is 
uncertainty in the body shape, electrode position and contact impedance. This means 
that a computed forward solution, based on an assumed conductivity (typically 
constant), has a much larger error than the errors inherent in the measurements. 
A compromise called difference imaging (by contrast to absolute imaging) uses a 
forward solution to calculate the linearization (25) and then forms an image of the 
difference of the conductivities between two different times, for example inspiration 
and expiration in a study of the lungs. Alternatively measurements can be taken 
simultaneously at two frequencies and a difference image formed of the permittivity. 

Given a basis of applied current patterns I; and a chosen set of measurements M; 
expressed as a set of independent vectors in S that are 1/|£;| for one electrode E;, 
—1/|E;| for another electrode Ex (the two electrodes between which the voltage 
is measured), and a set of functions yw; with the approximate admittivity satisfying 
Y = > ve We, the discretization of the Fréchet derivative is the Jacobian matrix 
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0 
Jiijyk = 5—Mj ZI; = -| WeVvi - Vuj dx, cn?) 
OK Q 


where L5u; = Lyv; = 0 (at least approximately) with u; satisfying the CEM 
with current I; and v; with M;. If the finite element approximation is used to solve 
the forward problem it has the interesting feature that the natural approximation 
to the Fréchet derivative in the FE context coincides with the Fréchet derivative 
of them FE approximation. The indices (ij) are bracketed together as they are 
typically “flattened” so the matrix of measurements becomes a vector and J a matrix 
(rather than a tensor). Let V be the vector of all voltage measurements, Visic the 
calculated voltages, and y the vector of y;. Our regularized least-squares version of 
the linearized problem is now 


Vreg = arg min||Sy — (V — Vea) ||? +o? U(y), (110) 
bg 


where W is a penalty function and a@ a regularization parameter. The same 
formulation is used for difference imaging where Veale is replaced by measured 
data at a different time or frequency. Typical choices for Y are quadratic penalties 
such a weighted sum of squares of the y;, the two norm of (a discretization) a 
partial differential operator R applied to y — yg, for some assumed background 
Yo- |[R (v — yo) ||?. Another common choice is a weighted sum of squares, i.e., L a 
positive diagonal matrix. In Total Variation regularization VY approximates ||V(y — 
Yo)||1, and can be used where discontinuities are expected in the conductivity. Where 
there is a jump discontinuity on a surface (a curve in the two-dimensional case) the 
total variation is the integral of the absolute value of the jump over the surface 
(curve). The choice of regularization parameter a, the choice of penalty function, 
and the solution methods are covered in the chapters » Linear Inverse Problems 
and “Total Variation in Imaging.” The singular values of J are found to decay 
faster than exponentially (see Fig. 9), so it is a severely illconditioned problem and 
regularization is needed even for very accurate data. There are also to some extent 
diminishing returns in increasing the number of electrodes without also increasing 
the measurement accuracy. 

For a quadratic penalty function the minimization problem with U(y) = ||R(y— 
Y)||7 the solution to (110) is given by the well-known Tikhonov inversion formula 


Vreg Yo = (IT + PR*R) | I* (V Vea) (111) 


For a total variation penalty Y(y) = ||R(y — yo)|{1 minimization is more 
difficult, and standard gradient based optimization methods have difficulty with the 
singularity in Y where a component of Ry vanishes. One way around this is to use 
the Primal Dual Interior Point Method; for details, see [21] and for comparison of 
TV and a quadratic penalty applied to a difference image of the chest, see Fig. 6. 
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Fig. 6 Time difference EIT image of a human thorax during breathing, comparison of generalized 
Tikhonov ||Ry||5 and of the TV ||Ry||; regularized algorithms. Both are represented on the same 
color scale and in arbitrary conductivity units. See [21] for details (a) Generalized Tikhonov (b) 
Total variation 


Regularized Iterative Nonlinear Methods 


As the problem is nonlinear clearly a solution based on linearization is inaccurate. 
Intuitively there are two aspects to the nonlinearity that are lost in a linear approxi- 
mation. If one considers an object of constant conductivity away from the boundary 
the norm of the voltage data will exhibit a sigmoid curve as the conductivity of 
that object varies, as seen in the example of a concentric anomaly and illustrated 
numerically in Fig. 10. This means that voltage measurements saturate, or tends to 
a limiting value, as the conductivity contrast to the background tends to zero or 
infinity. Typically this means that linear approximations underestimate conductivity 
contrast. One has to be careful in communications between mathematicians and 
engineers: the latter will sometimes take linearity (e.g., of Y(y)) to mean a function 
that is homogenous of degree one, ignoring the requirement for “superposition of 
solutions.” Considering two small spherical objects in a homogeneous background, 
it is known from (22) that to first order the change in u due to the objects is approx- 
imately the sum of two dipole fields. The effect of nonlinearity, the higher order 
terms in (22) can be thought of as interference between these two fields, analogous 
to higher order scattering in wave scattering problems. The practical effect is that 
linear approximations are poor not only at getting the correct conductivity contrast, 
but also at resolving a region between two objects that are close together. Many 
nonlinear solution methods calculate an update of the admittivity from solving a 
linear system, that update is applied to the conductivity in the model and the forward 
solution solved again. One severe problem with linear reconstruction methods that 
do not include a forward solver is that one cannot test if the updated admittivity even 
fits the data better than the initial assumption (e.g., of a constant admittivity). Such 
algorithms tend to produce some plausible image even if the data are erroneous. 
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The usual approach taken in geophysical and medical EIT to nonlinear recon- 
struction is to numerically perform the (nonlinear generalized Tikhonov) minimiza- 
tion 


Yreg = arg min||Veate(y) — VI? + a Y(y) (112) 
y 


using standard numerical optimization techniques. As the Jacobian is known 
explicitly it is efficient to use gradient optimization methods such as Gauss— 
Newton, and in that context the update step is very similar to the solution of 
the linear problem (110), and is a linear system for quadratic VY. Assuming 
conductivity initialized as the background level y, a typical iterative update scheme 
for successive approximations to the conductivity is 


Yn+1 = Yn a (Si In as oE*E) | (I30v ~~ Veaic(Y ») + wL* L(y, ~ Y»)) 

(113) 
In contrast to the linearized problem, the nonlinear problem requires repeated 
solution of the forward solution Veatc(Yn) for variable conductivity, typically 
using the finite element or finite difference method. One also has to constrain 
the conductivity Re y to be positive and this is made easier by a choice of ¢; 
as the characteristic functions of a partition on Q. This could be a rectangular 
grid or a courser tetrahedral mesh than that used for u. Accurate modeling of 
electrodes requires a fine discretization near electrodes, and yet one cannot hope 
to recover that level of detail in the admittivity near an electrode. In many practical 
situations a priori bounds are known for the conductivity and permittivity and as 
the logarithmic stability result predicts, enforcing these bounds has a stabilizing 
effect on the reconstruction. The positivity constraint can be enforced by a change 
of variables to logy and this is common practice, with the Jacobian adjusted 
accordingly. It is generally better to perform a line search in the update direction 
from (113) to minimize the cost function in (112) rather than simply applying the 
update. Most commonly this search is approximated, for example, by fitting a few 
points to a low order polynomial although implementation details of this are rarely 
well documented. It is also worth mentioning that most absolute reconstruction 
algorithms start by finding a homogeneous conductivity yg best fitting the data 
before the iterative method starts. 

In geophysical ERT nonlinear solution is well established. Although it is more 
common, for reasons of economy, to measure only along a line and reconstruct on 
the plane beneath that line, fully three-dimensional reconstruction is also widely 
used. The most common reconstruction code used is RES3DINV [44] which builds 
on the work of Loke and Barker at the University of Birmingham [84]. The code 
is available commercially from Loke’s company Geotomo Software. RES3DINV 
has a finite difference forward solver used when the ground is assumed flat, and 
a finite element solver for known non-flat topography. In geophysical applications 
there is the advantage that obtaining a trangularization of the surface is a common 
surveying practice. The Jacobian is initialized using an analytical initial solution 
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assuming homogeneous conductivity. Regularized nonlinear inversion is performed 
using Gauss—Newton, with recalculation of Jacobian [44], or using a quasi-Newton 
method in which a rank one update is performed on the Jacobian. The penalty 


function used in the regularization is of the form W(y) = ||Ry||3} where R is an 
approximate differential operator that penalizes horizontal and vertical variations 
differently. Total variation regularization V(y) = ||Ry||: is also an option in this 


code. When data is likely to be noisy one can select a “robust error norm,” in which 
the one-norm is used also to measure the fit of the data to the forward solution. A 
maximum and minimum value of the regularization parameter can be set by the user, 
but in a manner similar to the classical Levenburg—Marquard method for well-posed 
least squares problems the parameter can be varied within that range depending on 
the residual at each iteration. 

Although it is common in inverse problems to think of (112) as a regularization 
scheme a more rational justification for the method is probabilistic. Here, error 
in the measured data is considered to be sampled from a zero mean, possibly 
correlated, random variables. The a priori belief about the distribution of y is 
then represented as a probability distribution. The minimization (112) is the 
Maximum a posteriori (MAP) estimate for the case of independent Gaussian 
error and with prior distribution with log probability density proportional to 
—W(y). A more sophisticated approach goes beyond Gauss distributions and MAP 
estimates and samples the posterior distribution using Markov Chain Monte Carlo 
methods [69]. As this involves a large number of forward problem solutions, 
this is infeasible for large scale three-dimensional EIT problems. However as 
computers increase in speed and memory size relative to price, it is reasonable 
to expect this will eventually become a feasible approach. It will make it easier 
to approach EIT with a specific question such as “what is the volume of the 
region with a specified conductivity” with the answer expressed as an estimate of 
the probability distribution. Going back to (112) the regularization parameter a 
controls the ratio of the variances of the prior and error distribution. In practice 
this choice of this parameter is somewhat subjective, and the usual techniques 
in choice of regularization parameter, and the caution in their application, are 
relevant (Fig. 7). 

Results of a geophysical ERT study are shown in Fig. 8, thanks to the Geophys- 
ical Tomography Team, British Geological Survey (http://www.bgs.ac.uk/research/ 
tomography) for this figure and the description of the survey summarized below. 
In this case ERT was used to identify the concentrations of leachate, the liquid 
that escapes from buried waste in a landfill site. The leachate can be extracted and 
recirculated to enhance the production of landfill gas, which can ultimately be used 
for electricity generation. It was important to use a non-invasive technique — the 
more standard practice of drilling exploratory wells could lead to new flow paths. 
Data were collected sequentially on 64 parallel survey lines, using a regular grid 
of electrode positions. The inter-line spacing was 15 m with a minimum electrode 
spacing of 5 m along line. For the current sources, electrode spacings between 15 m 
and 95 m were used, while electrode spacings for the measurement electrodes were 
between 5 m and 225 m (Figs. 9 and 10). 
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Fig. 7 An “L-curve”: data mismatch ||V — Veaic(y)||2 (vertical) versus regularization norm 
||[R(v—yp)|l2 (Aorizontal) for a range of six orders of magnitude of the regularization. In each case, 
a single step of the iterative solution was taken. Three representative images are shown illustrating 
the “overregularization,” appropriate regularization, and “underregularization.” The data are from 
the RPI chest phantom [65] shown left 


Fig. 8 A three-dimensional ERT survey of a commercial landfill site to map the volumetric 
distribution of leachate (opaque blue). Leachate is abstracted and reproduction of any BGS 
materials does not amount to an endorsement by NERC or any of its employees of any product 
or service and no such endorsement should be stated or implied 
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Fig.9 Normalized singular values of the Jacobian matrix from circular 2D model with L = 
16, 24, and 32 electrodes. EIT measurements are made with trigonometric patterns such that the 
number of independent measurements from L electrodes is 3(L — 1)L. Note the use of more 
degrees of freedom in the conductivity than the data so as to be able to study the effect of different 


numbers of electrodes using SVD 
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Fig. 10 Saturation of EIT signals as a function of conductivity contrast. Left: Slices through a 
finite element model of a 2D circular medium with a circular conductivity target at four horizontal 
positions. EIT voltages are simulated at 32 electrodes for 31 trigonometric current patterns. Right: 
change in a voltage difference as a function of conductivity contrast (target vs. background) 
for each horizontal position (horizontal center of contrast specified in legend). Vertical axis is 
normalized with respect to the maximum change from the central target, and scaled by the sign of 
conductivity change 


The inversion was performed using RES3DINV with the FE forward solver with 
a mesh generated using the measured surface topography. The two norm was used 
for both penalty term and error norm. Due to the large number of datum points 
(approx 85,000 in total), the dataset was split in four approximately equal volumes 
for subsequent inversion. The resulting resistivity models were then merged to 
produce a final model for the entire survey area. The resulting 3D resistivity model 
was used to identify a total of 14 drilling locations for intrusive investigation. 
Eight wells were drilled and the results (i.e., initial leachate strikes within these 
wells) were used to calibrate the resistivity model. Based on this ground-truth 
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calibration a resistivity threshold value of 4 Qm was used to represent the spatial 
distribution of leachate for volumetric analysis within the waste mass. A commercial 
visualization package was used to display cross sections, iso-resistivity surfaces as 
well as topography and features on the surface and the boreholes. For other similar 
examples of geophysical ERT see [28, 29]. 

Experiments on tanks in process tomography or as simulated bodies for medical 
EIT show that several iterations of a nonlinear method can improve the accuracy 
of conductivity and the shape of conductivity contours for known objects. In 
medical EIT it has yet to be demonstrated that the shape and electrode position 
can be measured and modeled with sufficient accuracy that the error in the linear 
approximation is greater than the modeling error. These technical difficulties are 
not, hopefully, insurmountable (Fig. 11). 


Direct Nonlinear Solution 


Nachman’s [88, 89] (see also [94]) uniqueness result for the two-dimensional 
case was essentially constructive and has resulted in a family of reconstruction 
algorithms called d-methods or scattering transform methods. Nachman’s method 
was implemented by Siltanen et al. [100] in 2000. Of course there are few practical 
situations in which the two-dimensional approximation is a good one — both the 
conductivity and the electrodes have to be translationally invariant. Flow in a pipe 
with long electrodes is one example in which it is a good approximation. The main 
steps in the method are sketched (following Knudsen et al. [77]) and the interested 
reader referred to the references for details. 

Assuming Q2 is the unit disk for simplicity and starting with the Faddeev Green’s 
function 


eit & 
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Gis) = g(x), gel) = a : a € (114) 
R 
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Fig. 11 (continued) Iteration (horizontal axis) and regularization parameter selection (vertical 
axis) for two choices of regularization matrix R. Data are from the RPI chest phantom [65]. The 
regularization parameter [@ in (113)] in the middle row (1) was selected at the “knee” of the L- 
curve, indicating an appropriate level of regularization. Overregularization (top row) is shown for 
10a, and underregularization (bottom row) for 0.1a. Columns indicate 1, 3, or 5 iteration of (113). 
With increased iteration, one can see improved separation of targets and more accurate conductivity 
estimates, although these improvements trade off against increased electrode artifacts due to 
model mismatch. The difference between the Laplacian and weighted diagonal regularization is 
shown in the increased smoothness of (a), especially in the underregularized case. (a) Laplacian 
regularization. (b) Weighted diagonal regularization 
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Fig. 11 (continued) 
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and the single layer potential 


(S:@)(x) = : Gelx — y)6(y) 4009). (115) 
dQ 


Here k = k, + ik and, by abuse of notation, considering x as a vector in x -& 
and a complex number x; + ix2 in the complex product kx. Here 6(y) means the 
angular polar coordinate of y. Here, it is assumed that the measured Dirichlet-to- 
Neumann map A, is available and, of course, that A, is known. The first step in 
the algorithm is for each fixed k to solve the linear Fredholm integral equation for a 
function y(-,k) on the boundary. 


WC Alaa = el! — Se (Ay — AWC, Alan. (116) 


This is an explicit calculation of the Complex Geometrics Optics solution of 
Theorem 3. It is fed into the calculation of what is called the non-physical scattering 
transform t: C > C defined by 


t(k) = [eB a,-anwe.was, (117) 


dQ 


Note here that (116) is a linear equation to solve the resulting y depends nonlinearly 
on the data A, and of course as w depends on the data t is a nonlinear function of 
the data. The second step is to find the conductivity from the scattering data as 
follows. Let ex (k) := exp(i(kx + kx)). For each fixed x another integral equation 
is solved 


: ae e_x(k’) V(x, k) dki dk, (118) 


V(x,k) =1+ (27)? (k — k!)k! 
R2 


finally setting y(x) = V(x,0)*. The integral equation (118) is the solution to the 
partial differential equation 


dnV(x,k) = artllvelOVD. k eC, (119) 


where dx = 0/dk. Equation (119) is referred to as the 0 equation hence the name of 
the method. 

The reconstruction procedure is therefore a direct nonlinear method in which 
the steps are the solution of linear equations. The only forward modeling required 
is the construction of A,. In some practical realizations of this methods [65] an 
approximation to the scattering transform is used in which y is replaced by an 
exponential 
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t"(k) = [Fa —A,) dé. (120) 
dQ 


In practical reconstruction schemes t or t*P are replaced by an approximation 
truncated to zero for |k| > R for some R > 0, which effectively also truncates 
the domain of integration in (118) to the disk of radius R. Reconstruction of data 
from a two-dimensional agar phantom simulating a chest was performed in [65] 
using truncated t**?, and in [66] a difference imaging version of the d-method is 
implemented using a truncated scattering transform and applied to chest data. A 
rigorous regularization scheme for two-dimensional d-reconstruction is given in 
[77]. In this case the regularization is applied to the data, in a similar spirit to X-ray 
CT reconstruction in which the data is filtered and then backprojected (see chapter 

Tomography) and the regularization is applied in the filter on the data. In this 
sense it is harder to understand the regularized algorithm in terms of systematic a 
priori information applied to the image. As in CT this is traded off against having a 
fast explicit reconstruction algorithm that avoids iteration. 

So far the discussion of d-methods has been confined to two-dimensional 
problems. At the time of writing three-dimensional direct reconstruction methods 
are in their infancy. A three-dimensional d-algorithm for small conductivities is 
outlined in [34], and it is yet to be seen if this will result in practical implementation 
with noisy data on a finite array of electrodes. See the thesis of Bikowski [19] for 
the latest steps in this direction. If these efforts are successful, the impact on EIT is 
likely to be revolutionary. 


4 Conclusion 


Electrical impendence tomography and its relatives are among the most challenging 
inverse problems in imaging as the problem is nonlinear and highly illposed. The 
problem has inspired detailed theoretical and numerical study, and this has had an 
influence across a wide range of related inverse boundary value problems for (sys- 
tems of) partial differential equations. Medical and industrial process applications 
have yet to realize their potential as routine methods while the equivalent methods 
in geophysics are well established. A family of direct nonlinear solution techniques 
until recently only valid for the two-dimensional problem may soon be extended 
to practical three-dimensional algorithms. If this happens fast three-dimensional 
nonlinear reconstruction may be possible on relatively modest computers. In some 
practical situations in medical and geophysical EIT the conductivity is anisotropic, 
in which case the solution is non-unique. A specification of the a priori information 
needed for a unique solution is poorly understood and practical reconstruction 
algorithms have yet to be proposed in the anisotropic case. 

For a more complete summary of uniqueness results, the reader is referred to the 
review article of Uhlmann [110]. Similarly, for a review of biomedical applications 
of EIT, see the book by Holder [59], while subsequent progress in the medical area 
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can generally be found in special issues of the journal Physiological Measurement 
arising from the annual conferences on Biomedical Applications of EIT. A good 
reference for the details of geophysical EIT reconstruction can be found in the 
manual [44] and the notes by Loke [83]. For applications in process tomography 
see [117] and the proceedings of the biennial World Congress on Industrial Process 
Tomography (http://www. isipt.org/wcipt). 


References 


— 


. Adler, A., Lionheart, W.R.B.: Uses and abuses of EIDORS: an extensible software base for 


EIT. Physiol. Meas. 27, S25—S42 (2006) 


. Alessandrini, G.: Stable determination of conductivity by boundary measurements. Appl. 


Anal. 27, 153-172 (1988) 


. Alessandrini, G.: Singular solutions of elliptic equations and the determination of conductivity 


by boundary measurements. J. Differ. Equ. 84(2), 252-272 (1990) 


. Alessandrini, G.: Determining conductivity by boundary measurements, the stability issue. 


In: Spigler, R. (ed.) Applied and Industrial Mathematics, pp. 317-324. Kluwer, Dordrecht 
(1991) 


. Alessandrini, G.: Open issues of stability for the inverse conductivity problem. J. Inverse 


Ill-Posed Prob. 15, 451-460 (2007) 


. Alessandrini, G., Gaburro, R.: Determining conductivity with special anisotropy by boundary 


measurements. SIAM J. Math. Anal. 33, 153-171 (2001) 


. Alessandrini, G., Gaburro, R.: The local Calder6én problem and the determination at the 


boundary of the conductivity. Commun. PDE 34, 918-936 (2009) 


. Alessandrini, G., Vessella, S.: Lipschitz stability for the inverse conductivity problem. Adv. 


Appl. Math. 35, 207-241 (2005) 


. Ammari, H., Buffa, A., Nédélec, J.-C.: A justification of eddy currents model for the Maxwell 


equations. SIAM J. Appl. Math. 60, 1805-1823 (2000) 


. Astala, K., Paivarinta, L.: Calderén’s inverse conductivity problem in the plane. Ann. Math. 


163, 265-299 (2006) 


. Barber, D., Brown, B.: Recent developments in applied potential tomography — APT. In: 


Bacharach, S.L. (ed.) Information Processing in Medical Imaging, pp. 106-121. Nijhoff, 
Amsterdam (1986) 


. Barcelé, J.A., Faraco, D., Ruiz, A.: Stability of the inverse problem in the plane for less 


regular conductivities. J. Differ. Equ. 173, 231-270 (2001) 


. Barceld, J.A., Barcel6, T., Ruiz, A.: Stability of Calderon inverse conductivity problem in the 


plane. J. Math. Pure. Appl. 88, 522-556 (2007) 


. Beals, R., Coifman, R.: Transformation spectrales et equation d’evolution non lineares. 


Seminaire Goulaouic-Meyer-Schwarz 9, 1981-1982 (1982) 


. Beals, R., Coifman, R.R.: Linear spectral problems, non-linear equations and the d-method. 


Inverse Prob. 5, 87-130 (1989) 


. Beretta, E., Francini, E.: Lipschitz stability for the electrical impedance tomography problem: 


the complex case. Commun. PDE 36, 1723-1749 (2011) 


. Berenstein, C.A., Casadio Tarabusi, E.: Integral geometry in hyperbolic spaces and electrical 


impedance tomography. SIAM J. Appl. Math. 56, 755-64 (1996) 


. Beretta, E., de Hoop, M.V., Qiu, L.: Lipschitz stability of an inverse boundary value problem 


for a Schrédinger-type equation. SIAM J. Math. Anal. 45(2), 679-699 (2013) 


. Bikowski, J.: Electrical impedance tomography reconstructions in two and three dimensions; 


from Calderén to direct methods. Ph.D. thesis, Colorado State University, Fort Collins (2009) 


. Borcea, L.: Electrical impedance tomography. Inverse Probl. 18, R99-R136 (2002) [Borcea, 


L.: Addendum to electrical impedance tomography. Inverse Prob. 19, 997-998 (2003)] 


Electrical Impedance Tomography 759 


21 


22. 


23. 


24. 


25. 


26. 


27. 


28. 


29. 


30. 


31. 


32. 


33. 


34. 


35. 


36. 


37. 


38. 


39. 


40. 


41. 


42. 


43. 


44. 


. Borsic, A, Graham, B.M., Adler, A., Lionheart, W.R.B.: Total variation regularization in 
electrical impedance tomography. IEEE Trans. Med. Imaging 29(1), 44-54 (2010) 

Brown, R.: Global uniqueness in the impedance-imaging problem for less regural 
conductivities. SIAM J. Math. Anal. 27(4), 1049 (1996) 

Brown, R., Torres, R.: Uniqueness in the inverse conductivity problem for conductivities with 
3/2 derivatives in L?, p > 2n. J. Fourier Anal. Appl. 9, 1049-1056 (2003) 

Brown, R., Uhlmann, G.: Uniqueness in the inverse conductivity problem with less regular 
conductivities in two dimensions. Commun. PDE. 22, 1009-1027 (1997) 

Bukhgeim, A.L., Uhlmann, G.: Recovery a potential from partial Cauchy data. Commun. 
PDE. 27, 653-668 (2002) 

Calderén, A.P.: On an inverse boundary value problem. In: Seminar on Numerical Analysis 
and its Applications to Continuum Physics (Rio de Janeiro, 1980), pp. 65-73. Sociedade 
Brasileira de Matematica, Rio de Janeiro (1980) 

Calderén, A.P.: On an inverse boundary value problem. Comput. Appl. Math. 25(2-3), 
133-138 (2006) [Note this reprint has some different typographical errors from the original: 
in particular on the first page the Dirichlet data for w is @ not zero] 

Chambers, J.E., Meldrum, P.I., Ogilvy, R.D., Wilkinson, P.B.: Characterisation of a 
NAPL-contaminated former quarry site using electrical impedance tomography. Near Surf. 
Geophys. 3, 79-90 (2005) 

Chambers, J.E., Kuras, O., Meldrum, P.I., Ogilvy, R.D., Hollands, J.: Electrical resistivity 
tomography applied to geologic, hydrogeologic, and engineering investigations at a former 
waste-disposal site. Geophysics 71, B231—B239 (2006) 

Cheng, K., Isaacson, D., Newell, J.C., Gisser, D.G.:Electrode models for electric current 
computed tomography. IEEE Trans. Biomed. Eng. 36, 918-24 (1989) 

Cheney, M., Isaacson, D., Newell, J.C.: Electrical impedance tomography. SIAM Rev. 41, 
85-101 (1999) 

Ciulli, S., Ispas, S., Pidcock, M.K.: Anomalous thresholds and edge singularities in electrical 
impedance tomography. J. Math. Phys. 37, 4388 (1996) 

Colin de VerdiZere, Y., Gitler, I., Vertigan, D.: Réseaux électriques planaires. I. Comment. 
Math. Helv. 71, 144-167 (1996) 

Cornean, H., Knudsen, K., Siltanen, S.: Towards a D-bar reconstruction method for three 
dimensional EIT. J. Inverse Ill-Posed Prob. 14, 111-134 (2006) 

Di Cristo, M.: Stable determination of an inhomogeneous inclusion by local boundary 
measurements. J. Comput. Appl. Math. 198, 414-425 (2007) 

Dobson, D.C.: Stability and regularity of an inverse elliptic boundary value problem. 
Technical Report TR90-14 Rice University, Department of Mathematical Sciences (1990) 
Doerstling, B.H.: A 3-d reconstruction algorithm for the linearized inverse boundary value 
problem for Maxwell’s equations. Ph.D. thesis, Rensselaer Polytechnic Institute (1995) 
Druskin, V.: The unique solution of the inverse problem of electrical surveying and electrical 
well-logging for piecewise-constant conductivity. Izv. Earth Phys. 18, 51-53 (1982) (in 
Russian) 

Druskin, V.: On uniqueness of the determination of the three-dimensional underground struc- 
tures from surface measurements with variously positioned steady-state or monochromatic 
field sources. Sov. Phys.-Solid Earth 21, 210-214 (1985) (in Russian) 

Druskin, V.: On the uniqueness of inverse problems for incomplete boundary data. SIAM J. 
Appl. Math. 58(5), 1591-1603 (1998) 

Gaburro, R.: Sul problema inverso della tomografia da impedenza elettrica nel caso di 
conduttivita anisotropa, Tesi di Laurea in Matematica, Universita degli Studi di Trieste (1999) 
Gaburro, R.: Anisotropic conductivity inverse boundary value problems. Ph.D. thesis, 
University of Manchester Institute of Science and Technology (UMIST), Manchester (2003) 
Gaburro, R., Lionheart, W.R.B.: Recovering Riemannian metrics in monotone families from 
boundary data. Inverse Prob. 25, 045004 (2009) 

Geotomo Software: RES3DINV ver. 2.16, Rapid 3D Resistivity and IP Inversion Using the 
Least-Squares Method. Geotomo Software, Malaysia (2009). http://www.geoelectrical.com 


760 


45 


46. 
47. 


48. 


54. 


55. 


56. 


57. 


58. 


59. 


60. 


61. 


62. 


63. 


64. 


65. 


66. 


67. 


68. 


69. 


A. Adler et al. 


. Gisser, D.G., Isaacson, D., Newell, J.C.: Electric current computed tomography and 

eigenvalues. SIAM J. Appl. Math. 50, 1623-1634 (1990) 

Griffiths, H.: Magnetic induction tomography. Meas. Sci. Technol. 12, 1126-1131 (2001) 

Griffiths, H., Jossinet, J.: Bioelectric tissue spectroscopy from multi-frequency EIT. Physiol. 

Meas. 15(Suppl. 2A), 29-35 (1994) 

Haberman, B., Tataru, D.: Uniqueness in Calderén problem with Lipschitz conductivities. 

Duke Math. J. 162(3), 435-625 (2013) 

. Hahner, P.: A periodic Faddeev-type solution operator. J. Differ. Equ. 128, 300-308 (1996) 

. Hanke, M.: On real-time algorithms for the location search of discontinuous conductivities 
with one measurement. Inverse Prob. 24, 045005 (2008) 

. Hanke, M., Schappel, B.: The factorization method for electrical impedance tomography in 
the half-space. SIAM J. Appl. Math. 68, 907-924 (2008) 

. Hanke, M., Harrach, B., Hyndven, N.: Justification of point electrode models in electrical 
impedance tomography. Math. Models Methods Appl. Sci. 21, 1395 (2011) 

. Harrach, B., Seo, JK.: Exact shape-reconstruction by one-step linearization in electrical 

impedance tomography. SIAM J. Math. Anal. 42, 1505-1518 (2010) 

Harrach, B., Ullrich, M.: Monotonicity based shape reconstruction in electrical impedance 

tomography. SIAM J. Math. Anal. 45(6), 3382-3403 (2013). http://dx.doi.org/10.1137/ 

120886984 

Heck, H., Wang, J.-N.: Stability estimates for the inverse boundary value problem by partial 

Cauchy data. Inverse Prob. 22, 1787-1796 (2006) 

Heikkinen, L.M., Vilhunen, T., West, R.M., Vauhkonen, M.: Simultaneous reconstruction of 

electrode contact impedances and internal electrical properties: II. Laboratory experiments. 

Meas. Sci. Technol. 13, 1855 (2002) 

Heinrich, S., Schiffmann, H., Frerichs, A., Klockgether-Radke, A., Frerichs, I.: Body and head 

position effects on regional lung ventilation in infants: an electrical impedance tomography 

study. Intensive Care Med. 32, 1392-1398 (2006) 

Henderson, R.P., Webster, J.G.: An impedance camera for spatially specific measurements of 

the thorax. IEEE Trans. Biomed. Eng. BME-25(3), 250-254 (1978) 

Holder, D.S.: Electrical Impedance Tomography Methods History and Applications. 

Institute of Physics Publishing, Bristol (2005) 

Huang, S.M., Plaskowski, A., Xie, C.G., Beck, M.S.: Capacitance-based tomographic flow 

imaging system. Electron. Lett. 24, 418-419 (1988) 

Ikehata, M.: The enclosure method and its applications, Chapter 7. In: Analytic Extension 

Formulas and Their Applications (Fukuoka, 1999/Kyoto, 2000). International Society for 

Analysis, Applications and Computation, vol. 9, pp. 87-103. Kluwer Academic Publishers, 

Dordrecht (2001) 

Ikehata, M., Siltanen, S.: Numerical method for finding the convex hull of an inclusion in 

conductivity from boundary measurements. Inverse Prob. 16, 273-296 (2000) 

Ingerman, D., Morrow, J.A.: On a characterization of the kernel of the Dirichlet-to-Neumann 

map for a planar region. SIAM J. Math. Anal. 29, 106-115 (1998) 

Isaacson, D.: Distinguishability of conductivities by electric current computed tomography. 

IEEE Trans. Med. Imaging 5, 92-95 (1986) 

Isaacson, D., Mueller, J.L., Newell, J., Siltanen, S.: Reconstructions of chest phantoms by the 

d-bar method for electrical impedance tomography. IEEE Trans. Med. Imaging 23, 821-828 

(2004) 

Isaacson, D., Mueller, J.L., Newell, J., Siltanen, S.: Imaging cardiac activity by the D-bar 

method for electrical impedance tomography. Physiol. Meas. 27, S43-S50 (2006) 

Isakov, V.: Completeness of products of solutions and some inverse problems for PDE. J. 

Differ. Equ. 92, 305-317 (1991) 

Isakov, V.: On the uniqueness in the inverse conductivity problem with local data. Inverse 

Prob. Imaging 1, 95-105 (2007) 

Kaipio, J., Kolehmainen, V., Somersalo, E., Vauhkonen, M.: Statistical inversion and Monte 

Carlo sampling methods in electrical impedance tomography. Inverse Prob. 16, 1487-1522 

(2000) 


Electrical Impedance Tomography 761 


70 


71. 


72. 


73. 


74. 


wos 


76. 


7H, 


78. 


79, 


80. 


81. 


82. 


83. 


84. 


85. 


86. 


87. 


88. 


89. 


90. 


91. 


92. 


94. 


. Kang, H., Yun, K.: Boundary determination of conductivities and Riemannian metrics via 
local Dirichlet-to-Neumann operator. SIAM J. Math. Anal. 34, 719-735 (2002) 

Kenig, C., Salo, M.: Recent progress in the Calderén problem with partial data. Contemp. 
Math. 615, 193-222 (2014) 

Kenig, C., Salo, M.: The Calderén problem with partial data on manifolds and applications. 
Anal. PDE 6(8), 2003-2048 (2013) 

Kenig, C., Sjéstrand, J., Uhlmann, G.: The Calderén problem with partial data. Ann. Math. 
165, 567-591 (2007) 

Kim, Y., Woo, H.W.: A prototype system and reconstruction algorithms for electrical 
impedance technique in medical body imaging. Clin. Phys. Physiol. Meas. 8, 63-70 (1987) 
Kohn, R., Vogelius, M.: Identification of an unknown conductivity by means of measurements 
at the boundary. SIAM-AMS Proc. 14, 113-123 (1984) 

Kohn, R., Vogelius, M.: Determining conductivity by boundary measurements II. Interior 
results. Commun. Pure Appl. Math. 38, 643-667 (1985) 

Knudsen, K., Lassas, M., Mueller, J.L., Siltanen, S.: Regularized D-bar method for the inverse 
conductivity problem. Inverse Prob. Imaging 3, 599-562 (2009) 

Lassas, M., Uhlmann, G.: Determining a Riemannian manifold from boundary measurements. 
Ann. Sci. Ecole Norm. Sup. 34, 771-787 (2001) 

Lassas, M., Taylor, M., Uhlmann, G.: The Dirichlet-to-Neumann map for complete 
Riemannian manifolds with boundary. Commun. Geom. Anal. 11, 207-222 (2003) 

Lee, J.M., Uhlmann, G.: Determining anisotropic real-analytic conductivities by boundary 
measurements. Commun. Pure Appl. Math. 42, 1097-112 (1989) 

Lionheart, W.R.B.: Conformal uniqueness results in anisotropic electrical impedance imag- 
ing. Inverse Prob. 13, 125-134 (1997) 

Liu, L.: Stability estimates for the two-deimensional inverse conductivity problem. 
Ph.D. thesis, University of Rochester, New York (1997) 

Loke, M.H.: Tutorial: 2-D and 3-D electrical imaging surveys. Geotomo Software (2010). 
http://www.geoelectrical.com 

Loke, M.H., Barker, R.D.: Rapid least-squares inversion by a quasi-Newton method. Geo- 
phys. Prospect. 44, 131-152 (1996) 

Loke, M.H., Chambers, J.E., Ogilvy, R.D.: Inversion of 2D spectral induced polarization 
imaging data. Geophys. Prospect. 54, 287-301 (2006) 

Mandache, N.: Exponential instability in an inverse problem for the Schrédinger equation. 
Inverse Prob. 17, 1435-1444 (2001) 

Molinari, M., Blott, B.H., Cox, S.J., Daniell, G.J.: Optimal imaging with adaptive mesh 
refinement in electrical tomography. Physiol. Meas. 23(1), 121-128 (2002) 

Nachman, A.: Reconstructions from boundary measurements. Ann. Math. 128, 531-576 
(1988) 

Nachman, A.: Global uniqueness for a two dimensional inverse boundary value problem. Ann. 
Math. 143, 71-96 (1996) 

Nachman, A., Sylvester, J., Uhlmann, G.: An n-dimensional Borg-Levinson theorem. Com- 
mun. Math. Phys. 115, 593-605 (1988) 

Nakamura, G., Tanuma, K.: Local determination of conductivity at the boundary from the 
Dirichlet-to-Neumann map. Inverse Prob. 17, 405-419 (2001) 

Nakamura, G., Tanuma, K.: Direct determination of the derivatives of conductivity at the 
boundary from the localized Dirichlet to Neumann map. Commun. Korean Math. Soc. 16, 
415-425 (2001) 

. Nakamura, G., Tanuma, K.: Formulas for reconstrucing conductivity and its normal derivative 
at the boundary from the localized Dirichlet to Neumann map. In: Hon, Y.-C., Yamamoto, M., 
Cheng, J., Lee, J.-Y. (eds.) The First International Conference on Inverse Problems: Recent 
Development in Theories and Numerics, City University of Hong Kong, Jan 2002, pp. 192- 
201. World Scientific, River Edge (2002) 

Novikov, R.G.: A multidimensional inverse spectral problem for the equation —Ay + 
(v(x) — Eu(x))v = 0. Funktsional. Anal. i Prilozhen. 22(4), 11-22, 96 (1988) (in Russian) 
[translation in Funct. Anal. Appl. 22(1988), no. 4, 263-272 (1989)] 


762 


95 


96 


97 


98 


99 


100. 


101. 


102. 


103. 


104. 


105. 


106. 


107. 


108. 


109. 


110. 


111. 


112. 


113. 


114. 


115. 


116. 


117. 


A. Adler et al. 


. Paivarinta, L., Panchenko, A., Uhlmann, G.: Complex geometrical optics solutions for 
Lipschitz conductivities. Rev. Mat. Iberoamericana 19, 57—72 (2003) 

. Paulson, K., Breckon, W., Pidcock, M.: Electrode modeling in electrical-impedance 
tomography. SIAM J. Appl. Math. 52, 1012-1022 (1992) 

. Polydorides, N., Lionheart, W.R.B.: A Matlab toolkit for three-dimensional electrical 
impedance tomography: a contribution to the electrical impedance and diffuse optical 
reconstruction software project. Meas. Sci. Technol. 13, 1871-1883 (2002) 

. Seagar, A.D.: Probing with low frequency electric current. Ph.D. thesis, University of 
Canterbury, Christchurch (1983) 

. Seagar, A.D., Bates, R.H.T.: Full-wave computed tomography. Pt 4: low-frequency electric 

current CT. Inst. Electr. Eng Proc. Pt. A 132, 455-466 (1985) 

Siltanen, S., Mueller, J.L., Isaacson, D.: An implementation of the reconstruction algorithm 

of A. Nachman for the 2-D inverse conductivity problem. Inverse Prob. 16, 681-699 (2000) 

Soleimani, M., Lionheart, W.R.B.: Nonlinear image reconstruction for electrical capacitance 

tomography experimental data using. Meas. Sci. Technol. 16(10), 1987-1996 (2005) 

Soleimani, M., Lionheart, W.R.B., Dorn, O.: Level set reconstruction of conductivity and 

permittivity from boundary electrical measurements using expeimental data. Inverse Prob. 

Sci. Eng. 14, 193-210 (2006) 

Somersalo, E., Cheney, M., Isaacson, D.: Existence and uniqueness for electrode models for 

electric current computed tomography. SIAM J. Appl. Math. 52, 1023-1040 (1992) 

Soni, N.K.: Breast imaging using electrical impedance tomography. Ph.D. thesis, Dartmouth 

College (2005) 

Sylvester, J.: An anisotropic inverse boundary value problem. Commun. Pure. Appl. Math. 

43, 201-232 (1990) 

Sylvester, J., Uhlmann, G.: A uniqueness theorem for an inverse boundary value problem in 

electrical prospection. Commun. Pure Appl. Math. 39, 92-112 (1986) 

Sylvester, J., Uhlmann, G.: A global uniqueness theorem for an inverse boundary valued 

problem. Ann. Math. 125, 153-169 (1987) 

Sylvester, J., Uhlmann, G.: Inverse boundary value problems at the boundary-continuous 

dependence. Commun. Pure Appl. Math. 41, 197-221 (1988) 

Tamburrino, A., Rubinacci, G.: A new non-iterative inversion method for electrical resistance 

tomography. Inverse Prob. 18, 1809-1829 (2002) 

Uhlmann, G.: Topical review: electrical impedance tomography and Calderén’s problem. 

Inverse Prob. 25, 123011 (2009) 

Vauhkonen, M.: Electrical impedance tomography and prior information. Ph.D. thesis, 

University of Kuopio (1997) 

Vauhkonen, M., Karjalainen, P.A., Kaipio, J.P.: A Kalman filter approach to track fast 

impedance changes in electrical impedance tomography. IEEE Trans. Biomed. Eng. 45, 

486-493 (1998) 

Vauhkonen, M., Lionheart, W.R.B., Heikkinen, L.M., Vauhkonen, PJ., Kaipio, J.P: 

A MATLAB package for the EIDORS project to reconstruct two-dimensional EIT images. 

Physiol. Meas. 22, 107-111 (2001) 

West, R.M., Jia, X., Williams, R.A.: Parametric modelling in industrial process tomography. 

Chem. Eng. J. 77, 31-36 (2000) 

West, R.M., Soleimani, M., Aykroyd, R.G., Lionheart, W.R.B.: Speed improvement of 

MCMC image reconstruction in tomography by partial linearization. Int. J. Tomogr. Stat. 

4(S06), 13-23 (2006) 

Yang, W.Q., Spink, D.M., York, T.A., McCann, H.: An image-reconstruction algorithm based 

on Landweber’s iteration method for electrical-capacitance tomography. Meas. Sci. Technol. 

10, 1065-1069 (1999) 

York, T.: Status of electrical tomography in industrial applications. J. Electron. Imag. 10, 608 

(2001) 


Synthetic Aperture Radar Imaging 


Margaret Cheney and Brett Borden 


Contents 


DA Gs x ee srcthiens wie salhestedaition vafelsinea’s cdaalecapientenrninaa va dacarann abet aulecuabi cates, 
Hastoricall Back erouid 3.00.05 cosaiasuansis cs cas ceaontads ceeaacel a nade dns ate cee Meee eS 
S- - Mathematical Modeling so... :..ccissnnedsnceieeenswsnnesacneasaedsonenennasesioneconmeasaiines 
HCaeeriny GF FlechOMagMEe WAVER osc acacienand ate casacawatna td soevaeaiasaneiguies 
Basie Facts About the Wave Rqnation. occ ccccceecencces cea cone auwer edie se saeeseenees 
Basie Seaiterine Mey es ciecyiy sawties cas sylenwnas socteneyenveatann anes quan syobenhy Sues 
NU Vee Ute DCN sic sis or sicisla siete slaisls oie Soi wip seis alg inn ccetcia cersla diaslng ctoangedieanigasciatnapetes 
Model Sor He Scattered Pleld. ooo... soon enasievesinn sieve ee eiinesninecncieisinaeiaisnaines oeimasiein ses 
Vile Matec Gamat. ce ye ccesccetis ond neue spas salen Melednchiaaan caus dans dagrealteyiannemetes 
‘The Small-SGetie A poron ination «035 tases cs cnkantads elaawnacaeaccnanecas donate cae ee ce 
The Rane) Prone... cic cchwncns cesmncesendensaedwinadunndne desman hendibaceswiaars muewedes 
4 Survey on Mathematical Analysis of Methods................ 0.0 cece cece eee eee eee e eee 
Inverse Synthetic Aperture Radar (ISAR).........00s00ssecsecessacs seecnesseenesecnensevies 
Poy MUTE PITTS RAGE te sco cle cbaenarmatiannice smedua cua ens Sledes bin die wcatie dence cie’ 
Resolution tor [SAR arid Spotlight SAR occ. scccceccaseses sctues caunwsesaeedaus eee eee 
S Nuria WeMHOdS oi j5..cs.0.c.coaireswanadoard amiicne nae gains reeaebanns eis sha raymimend soem eiones 
ISAR and Spotlight SAR Alsorithims: ¢.0:.ccssccsncdesicgiean dese eda cer iacaka ceassonvanies 
Range Alamein: os is cas seaawerceesicag cease cuseen neta sauas tauweduauatenareesenueeees 
G:C PRINS saints, so aiadess Bercinrsticistinascig Saracens nualnnsteindicisie meters ccna dwiaals degen sa waete & 
Problems Kelated to Unmodeled Motion... .:csccc case ccie eee dete asteaeationsaiooaei davies 
Problems Related to Unmodeled Scattering Physics.................c ccs eee cece eee e es 
New Applications of Radar Gmagiteg...0 06.45 c0 cnc ccsencaesnisaceevesteaemeanasaeenenaces 
2 AMNION occa wcncaeancenersame chap anus deunsacgnanceanectaseeennedeuas tina mtammccaee tis 
CHGGS-RSTSPOTIOER iccs os cine donned amacnionnpeminnaeameaninetyoainaneteaee oon neawiodes@nmate dns tmedse ned 
BOPeLCHORS oes ess aisie aches teeter sucnacne des Oi del phee ea dechataapatcepnldisdclatee see eabgiuinneacees 


M. Cheney (2) 

Department of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA 
e-mail: cheney @rpi.edu 

B. Borden 


Department of Physics, Naval Postgraduate School of Engineering, Monterey, CA, USA 
e-mail: bhborden@nps.edu 


© Springer Science+Business Media New York 2015 
O. Scherzer (ed.), Handbook of Mathematical Methods in Imaging, 
DOI 10.1007/978-1-4939-0790-8_49 


763 


764 M. Cheney and B. Borden 


Abstract 

The purpose of this chapter is to explain the basics of radar imaging and 
to list a variety of associated open problems. After a short section on the 
historical background, the chapter includes a derivation of an approximate scalar 
model for radar data. The basics in inverse synthetic aperture radar (ISAR) are 
discussed, and a connection is made with the Radon transform. Two types of 
synthetic aperture radar (SAR), namely, spotlight SAR and stripmap SAR, are 
outlined. Resolution analysis is included for ISAR and spotlight SAR. Some 
numerical algorithms are discussed. Finally, the chapter ends with a listing of 
open problems and a bibliography for further reading. 


1 Introduction 


“Radar” is an acronym for RAdio Detection And Ranging. Radar was originally 
developed [7, 8, 64, 67, 72] as a technique for detecting objects and determining 
their positions by means of echolocation, and this remains the principal function 
of modern radar systems. However, radar systems have evolved over more than 
seven decades to perform an additional variety of very complex functions; one such 
function is imaging [9, 20-22, 26, 29, 35,41,59, 61]. 

Radar-based imaging is a technology that has been developed mainly within 
the engineering community. There are good reasons for this: some of the critical 
challenges are (1) transmitting microwave energy at high power, (2) detecting 
microwave energy, and (3) interpreting and extracting information from the received 
signals. The first two problems are concerned with the development of appropriate 
hardware; however, these problems have now largely been solved, although there 
is ongoing work to make the hardware smaller and lighter. The third problem 
essentially encompasses a set of mathematical challenges, and this is the area where 
most of the current effort is taking place. 

Radar imaging shares much in common with optical imaging: both processes 
involve the use of electromagnetic waves to form images. The main difference 
between the two is that the wavelengths of radar are much longer than those of 
optics. Because the resolving ability of an imaging system depends on the ratio of 
the wavelength to the size of the aperture, radar imaging systems require an aperture 
many thousands of times larger than optical systems in order to achieve comparable 
resolution. Since kilometer-sized antennas are not practicable, fine-resolution radar 
imaging has come to rely on the so-called synthetic apertures in which a small 
antenna is used to sequentially sample a much larger measurement region. 


2 Historical Background 


Radar technology underwent rapid development during World War II; most of this 
work concerned developing methods to transmit radio waves and detect scattered 
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waves. The invention of synthetic aperture radar (SAR) is generally credited to 
Carl Wiley, of the Goodyear Aircraft Corporation, in 1951. The mid-1950s saw 
the development of the first operational systems, under the sponsorship of the US 
Department of Defense. These systems were developed by a collaboration between 
universities, such as the University of Illinois and the University of Michigan, 
together with companies such as Goodyear Aircraft, General Electric, Philco, and 
Varian. In the late 1960s, the National Aeronautics and Space Administration 
(NASA) began sponsoring unclassified work on SAR. Around this time, the first 
digital SAR processors were developed (earlier systems having used analogue 
optical processing). In 1978, the SEASAT-A satellite was sent up, and even though 
it operated only for 100 days, the images obtained from it were so useful that it 
became obvious that more such satellites were needed. In 1981, the shuttle imaging 
radar (SIR) series began, and many shuttle missions since then have involved radar 
imaging of the earth. In the 1990s, satellites were sent up by many countries 
(including Canada, Japan, and the European Space Agency), and SAR systems 
were sent to other planets and their moons, including Venus, Mars, and Titan. 
Since the beginning of the new millennium, more satellites have been launched, for 
example, the new European Space Agency satellite ENVISAT and the TerraSAR-X 
satellite, which was developed and launched by a (mainly European) public—private 
partnership. 

Code letters for the radar frequency bands were originally used during wartime, 
and the usage has persisted. These abbreviations are listed in Table |. The HF band 
usually carries radio signals; VHF carries radio and broadcast television; the UHF 
band carries television, navigation radar, and cell phone signals. Some radar systems 
operate at VHF and UHF; these are typically systems built for penetrating foliage, 
soil, and buildings. Most of the satellite synthetic aperture radar systems operate in 
the L-, S-, and C-bands. The S-band carries wireless Internet. Many military systems 
operate at X-band. 


Table 1 Radar frequency bands 


Band designation Approximate frequency range Approximate wavelengths 
HF (high frequency) 3-30 MHz 50m 
VHF (very high frequency) 30-300 MHz Sm 
UHF (ultrahigh frequency) 300-1,000 MHz 1m 
L-band 1-2 GHz 20cm 
S-band 2-4 GHz 10cm 
C-band 4-8 GHz Scm 
X-band 8-12 GHz 3cm 
Ku-band (under K) 12-18 GHz 2cm 
K-band 18-27 GHz 1.5cm 
Ka-band (above K) 27-40 GHz lcm 


mm-wave 40-300 GHz 5mm 
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3 Mathematical Modeling 


SAR relies on a number of very specific simplifying assumptions about radar 
scattering phenomenology and data-collection scenarios: 


1. Most imaging radar systems make use of the start—-stop approximation [29], in 
which both the radar sensor and scattering object are assumed to be stationary 
during the time interval over which the pulse interacts with the target. 

2. The target or scene is assumed to behave as a rigid body. 

3. SAR imaging methods assume a linear relationship between the data and scene. 


Scattering of Electromagnetic Waves 


The present discussion considers only scattering from targets that are stationary. 
For linear materials, Maxwell’s equations can be used [34] to obtain an inhomo- 
geneous wave equation for the electric field € at time ¢ and position x: 


2 
| a E(t, x) = s(t,x) (1) 


Vv? - = 
ee) c2(x) ar? 


and a similar equation for the magnetic field B. Here c(x) denotes the speed of 
propagation of the wave (throughout the atmosphere, this speed is approximately 
independent of position and equal to the constant vacuum speed c) and s is a source 
term that, in general, can involve € and B. For typical radar problems, the wave 
speed is constant in the region between the source and the scattering objects (targets) 
and varies only within the target volume. Consequently, here scattering objects are 
modeled solely via the source term s(t, x). 

One Cartesian component of Eq. (1) is 


2, 
(7 nse ) £0.2) = s(e2, (2) 


2 ar? 


where atmospheric propagation between source and target has been assumed. 


Basic Facts About the Wave Equation 


A fundamental solution [69] of the inhomogeneous wave equation is a generalized 
function [30, 69] satisfying 


, «1 
(v - =3) g(t,x) = —8(t)6(e). (3) 


The solution of (3) that is useful is 
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8(t — [xl/c) / e“ioe-tl/e) 
t,x) = = da, 4 
g(x) Ar |x| 827 |x| 
where in the second equality the identity 
1 —iot 
6(t) = — Je da (5) 
20 


was used. The function g(t, x) can be physically interpreted as the field at (t, x) due 
to a source at the origin x = 0 at time t = 0 and is called the outgoing fundamental 
solution or (outgoing) Green’s function. 

The Green’s function [62] can be used to solve the constant-speed wave equation 
with any source term. In particular, the outgoing solution of 


(v° a =) u(t,x) = s(t,x) (6) 
c? a? ; eG 
1S 
u(t,x) = - | g(t —t’,x —y)s(t’,y) dt’ dy. (7) 


In the frequency domain, the equations corresponding to (3) and (4) are 


elk |x! 


(V7 +k7)G=-6 and G(w,x) = . 
An |x| 


(8) 


where the wave number k is defined as k = w/c. 


Basic Scattering Theory 


In constant-wave velocity radar problems, the source s is a sum of two terms, s = 
s'" + s°°, where si models the transmitting antenna and s*° models the scattering 
object. The solution € to Eq. (1), which is written as €'", therefore splits into two 
parts: € = €"+4€*. The first term, E'", satisfies the wave equation for the known, 
prescribed source s'". This part is called the incident field, because it is incident upon 
the scatterers. The second term, E*, is due to target scattering, and this part is called 
the scattered field. We use the same decomposition in the simplified scalar model. 
One approach to finding the scattered field is to simply solve (2) directly, using, 
for example, numerical time-domain techniques. For many purposes, however, it is 
convenient to reformulate the scattering problem in terms of an integral equation. 
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The Lippmann-Schwinger Integral Equation 

In scattering problems the source term s* (typically) represents the target’s response 
to an incident field. This part of the source function will generally depend on the 
geometric and material properties of the target and on the form and strength of the 
incident field. Consequently, s*° can be quite complicated to describe analytically, 
and in general, it will not have the same direction as si. Fortunately, for this 
article, it is not necessary to provide a detailed analysis of the target’s response; 
for stationary objects consisting of linear materials, the scalar model s*° is written 
as the time-domain convolution 


s*(t,x) = [ree neras dt’, (9) 


where v(f, x) is called the reflectivity function and depends on target orientation. In 
general, this function also accounts for polarization effects. 

The expression (9) is used in (7) to express E*° in terms of the Lippmann— 
Schwinger integral equation [47] 


E“(t,x) = [ —1,x—2) // v(t — t',z)E™(t’,z) dt’ dr dz. (10) 


The Lippmann-Schwinger Equation in the Frequency Domain 
In the frequency domain, the electric field and reflectivity function become 


E(o.x) = f e@te(.xyat and Viw.z) = f eo"v(z) at (11) 


respectively. Thus the frequency-domain version of (2) is 


2 
(v° + -) E(w,x) = S(@,x) (12) 
and of (10) is 
E“(@,x) = -f G(o,x —z)V(o,z) E(w, z) dz. (13) 


The reflectivity function V(w,x) can display a sensitive dependence on w 
[34, 36, 53]. When the target is small in comparison with the wavelength of the 
incident field, for example, V is proportional to w? (this behavior is known as 
“Rayleigh scattering”). At higher frequencies (shorter wavelengths), the dependence 
on @ is typically less pronounced. In the so-called optical region, V(w,x) is 
often approximated as being independent of w (see, however, [56]); the optical 
approximation is used in this chapter, and the w dependence is simply dropped. 


Synthetic Aperture Radar Imaging 769 


In the time domain, this corresponds to v(t,z) = 45(t)V(z), and the delta function 
can be used to carry out the f’ integration in (10). 


The Born Approximation 

For radar imaging, the field €*° is measured at the radar antenna, and from these 
measurements, the goal is to determine V. However, both V and €*° in the 
neighborhood of the target are unknown, and in (10) these unknowns are multiplied 
together. This nonlinearity makes it difficult to solve for V. Consequently, almost 
all work on radar imaging relies on the Born approximation, which is also known 
as the weak-scattering or single-scattering approximation [38, 47]. The Born 
approximation replaces €'* on the right side of (10) by €, which is known. This 
results in a linear formula for €°° in terms of V: 


E°(t,x) ~ Ep(t,x) = // g(t — t,x —z)V(z)E™(t,z) dt dz. (14) 


In the frequency domain, the Born approximation is 


ik|x—z| 


EX(@,x) = -| V(z)E'"(@,z) dz. (15) 


4n|x —z| 


The Born approximation is very useful because it makes the imaging problem linear. 
It is not, however, always a good approximation; see Sect. 6. 


The Incident Field 


The incident field € is obtained by solving (2), where s'" is taken to be the relevant 
component of the current density on the source antenna and s*° is zero. This article 
uses a simplified point-like antenna model, for which si"(t,x) = p(t)d(x — x°), 
where p is the waveform transmitted by the antenna. Typically p consists of a 
sequence of time-shifted pulses, so that p(t) = )° po(t — tn). 

In the frequency domain, the corresponding source for (12) is S'"(@,x) = 
P(w)5(x — x°), where P denotes the inverse Fourier transform of p: 


p(t) = oe | eet Pw) do. (16) 
Qn 


Use of (8) shows that the incident field in the frequency domain is 


E"(,x) = -{ G(o,x—y)P(w)8(y — 2) ay 
eiklx—x?| 


yn 
is — x9| 


(17) 
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Model for the Scattered Field 


In monostatic radar systems, the transmit and receive antennas are colocated — often 


the same antenna is used. Use of (17) in (15) shows that the Born-approximated 


scattered field at the transmitter location x° is 


e2ik |x°—z| 


Fourier transforming (18) results in an expression for the time-domain field: 
e ielt— —2|x°—z|/c) 
t,x —————.; P(w) V(z) daw dz 
Ex (x) = Il Seam 2 (47|x9 — z|)? eve) 


= [i ~2\/c) 


(An|x? = 2))* 


V(z) dz. (19) 


Under the Born approximation, the scattered field is a superposition of scattered 
fields from point-like targets V(z’) « 6(z—2z’). 


The Matched Filter 


An important aspect of (19) is the 1/R* geometrical decay (where R = |x° —z)). 
When R is large (which it usually is), this decay factor results in a received signal 
that is extremely small — so small, in fact, that it can be dominated by thermal noise 
in the receiver. Thus it is difficult even to detect the presence of a target. Target 
detection is typically accomplished by means of a matched filter [19, 25,50]. 

Below the matched filter is derived for scattering from a single fixed, point-like 
target. For such a target, by Eqs. (9) and (19), the signal scattered is simply a time- 
delayed version of the transmitted waveform: 


Stec(t) = ps(t — tT) + n(t), 


where t corresponds to the 2R/c delay, p is a proportionality factor related to the 
scatterer reflectivity V(z) and the geometric decay (4z|x° — z|)~?, and n denotes 
noise. 

The strategy is to apply a filter (convolution operator) to Syec in order to improve 
the signal-to-noise ratio. The filter’s impulse response (convolution kernel) is 
denoted by h, which implies that the filter output is 


n(t) = (h * Stec)(t) = pns(t) + Nn (t), (20) 


where 
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ns(t) = [re —t')s(t’—1)dt’ and n,(t)= fre-tone dt’. 


The signal output 7,;(t) at time t should be as large as possible, relative to the noise 
output 7, (T). 

The noise is modeled as a random process. Thermal noise in the receiver is well 
approximated by white noise, which means that (n(t)n*(t’)) = Né(t — t’), where 
N corresponds to the noise power and (-) denotes expected value. Since the noise 
is random, so is 7,,. Thus the signal-to-noise (SNR) ratio to be maximized is 


2 
SNR = lis (21) 


(Inn (t)|?) 


] 
= (icc — t/)n(t’) de’ (fre —t"\n(t’) a") | 


= | h(t —t')h* (ct — t”) (n(t’)n* (t")) de’ dt” 
— —-——— 


N6(t'/—t’’) 


= f |ne=29P a’ =n f |neyPar, 


First, the denominator of (21) is 


(In(2)P) = ( : h(t —t'yn(1') da 


where in the last line the change of variables tf = t — t’ has been made and where 
the star denotes complex conjugation. Thus (21) becomes 


[ [ 


_ |fh(e—t’)s(t’—r) dt’ | f h(e)s(—t) de 


SNR="NJlkOR a ONS Th@ Par 


(22) 


where in the numerator the change of variables t = t — t’ has been made. To the 
numerator of (22), the Cauchy—Schwarz inequality can be used to conclude that the 
numerator, and therefore the quotient (22), is maximized when h is chosen, so that 


h(t) = s*(-t). 


This is the impulse response of the matched filter. Thus to obtain the best signal- 
to-noise ratio, the received signal should be convolved with the time-reversed, 
complex-conjugated version of the expected signal. 

With this choice, the filter (20) can be written as 
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n(t) = / h(t—t")see(t) dt” = , s*(0"—1)s0(t") dt" = i S*()Syo(t! +0) dt" 

(23) 
which is a correlation between § and Sjec. If S = Stee, (23) is called an autocorre- 
lation. Radar receivers which perform this kind of signal processing are known as 
“correlation receivers.” 


The Effect of Matched Filtering on Radar Data When applied to (18), the output 
of the correlation receiver is 


n(t,x°) ~ [ore -neg@.a%ar 


1 ia! (t!—2) j ew iolt ’—2|x°—z|/c) ere 
-{(5 Je P*(a’) ’) (a ae owdz dt 


ei olt— 2|x9—z|/c) 
- {f= railed i(w—w)t’ dt’ ——_—_—_ P(w) P* (w') V(z)da'dwdz 


(4|x9 —z})2- 
joo 
eio(t—2|x°—z|/c) 
- {> Ga? Sa) ——_____|P(w)|*V(z)dwdz. (24) 


Thus, the effect of matched filtering is simply to replace P(@) in the first line of (19) 
by 2z|P(o)|’. 


The Small-Scene Approximation 


The small-scene approximation, namely, 


2 
Ix—y| = ni-ty+0(7), (25) 


where x denotes a unit vector in the direction x and is often applied to situations in 
which the scene to be imaged is small in comparison with its average distance from 
the radar. This approximation is valid for |x| >> ly]. 

Use of (25) in (4) gives rise to the large-|x| expansion of the Green’s function: 
[12, 19] 


ik|x—y| ik|x| oe k 2 
G(w,x-y) =~ = © (ip 0())\ (14 0(@ 
An|x—y|  4ac|x| [x] [xl os 


Here, the first-order term must be included in the exponential because kx -y can on 
on values that are large fractions of 27. 
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Small-Scene, Matched-Filtered Radar Data In (19), the origin of coordinates can 
be chosen to be in or near the target, and then the small-scene expansion (26) (with 
z playing the role of y) can be used in the matched-filtered version of (19). This 
results in the expression for the matched-filtered data: 


1 
(470)? |[x°|? 


np(t) = // ein t21x°|/e+2% -2/e)) P(e) 2 V(z) dew dz. (27) 


The inverse Fourier transform of (27) gives 


e2ik|x?| . ve 
Dz(@) = Gayppoe P| le "ZV (z) dz. (28) 
——— 
FIV|2k2°) 


Thus we see that each frequency component of the data provides us with a Fourier 
component of the reflectivity V. 


The Range Profile 


Signals with large bandwidth are commonly used in synthetic aperture imaging. 
When the bandwidth is large, the pulse p is said to be a high-range-resolution 
(HRR) pulse. An especially simple large-bandwidth signal is one for which | P(w)|* 
is constant over its support. In this case, the w-integral in Eq. (27) reduces to a scaled 
sinc(t) function centered on 


t = 2\x°|/e + 2%°-z/c, 
and the width of this sinc function is inversely proportional to the bandwidth. When 


the support of |P(q)|* is infinite, of course, this sinc(t) becomes a delta function. 
Thus large-bandwidth (HRR), matched-filtered data can be approximated by 


1 x 
np(t) © aap | 80 — 2° + 29° 2/e)V Cae (29) 


Since time delay and range are related in monostatic radar systems as tf = 2R/c, 
Eq. (29) can be seen to be a relation between the radar data ng (¢) and the integral of 
the target reflectivity function over the plane 


R= |x| +2°-z 


(with respect to the radar). Such data are said to form a “range profile” of the target. 
An example of an HRR range profile is displayed in Fig. |. 
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Fig. 1 Example of an HRR range profile of an aircraft (orientation displayed in inser) 


4 Survey on Mathematical Analysis of Methods 


The mathematical models discussed above assume that the target V(z) is stationary 
during its interaction with a radar pulse. However, synthetic aperture imaging 
techniques assume that the target moves with respect to the radar between pulses. 


Inverse Synthetic Aperture Radar (ISAR) 


A fixed radar system staring at a rotating target is equivalent (by change of 
reference frame) to a stationary target viewed by a radar moving (from pulse 
to pulse) on a circular arc. This circular arc will define, over time, a synthetic 
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aperture, and sequential radar pulses can be used to sample those data that would 
be collected by a much larger radar antenna. Radar imaging based on such a 
data-collection configuration is known as inverse synthetic aperture radar (ISAR) 
imaging [5, 15, 41,59, 66,74]. This imaging scheme is typically used for imaging 
airplanes, spacecraft, and ships. In these cases, the target is relatively small and 
usually isolated. 


Modeling Rotating Targets The target reflectivity function in a frame fixed to 
the target is denoted by q. Then, as seen by the radar, the reflectivity function is 
V(x) = q(O(6,)x), where O is an orthogonal matrix and where ¢, = 0, denotes 
the time at the start of the nth pulse of the sequence. 

For example, if the radar is in the plane perpendicular to the axis of rotation 
(so-called turntable geometry), then the orthogonal matrix O can be written as 


cos @ —sin6 0 
O(@) = | sin@ cosé 0 (30) 
0 Oo 1 


and V(x) = g(x, cos 8 — x2 sin 0, x; sin @ + x2 cos 6, x3). 


Radar Data from Rotating Targets The use of Vix) = q(O(6,)x) in (28) 
provides a model for the data from the nth pulse: 


‘ 0 
e2ik \x°| 


Dolo.) = Gome—rsl Pole)? / ek +24 (2(6,,)z) dz. G1) 
y 


In (31), the change of variables y = O(6,)z is made. Then use is made of the 
fact that the inverse of an orthogonal matrix is its transpose, which means that 
x°-O-!(6,)y = O(6,,)x° -y. The result is that (31) can be written in the form 


2ik |x| 


D3 (@, ) = (4x22 


i 0, 
| Po(w)|? / e HOO “Ia (y) dy . (32) 
ae” 


OF [g\(2kO (0, )%°) 


Thus, the frequency-domain data are proportional to the inverse Fourier transform 
of g, evaluated at points in a domain defined by the angles of the sampled target 
orientation and the radar bandwidth (see Fig. 2). Consequently, a Fourier transform 
produces a target image. 

The target rotation angle is usually not known. However, if the target is rotating 
with constant angular velocity, the image produced by the Fourier transform gives 
rise to a stretched or contracted image, from which the target is usually recognizable 
[5,41, 66, 72]. 
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Fig. 2 The data-collection 
manifold for turntable 
geometry 


The Data-Collection Manifold 
The Fourier components of the target that can be measured by the radar are those in 
the set 


Qz = {2kO(O,)*°%, (33) 


where n ranges over the indices of pulses for which the point z is in the antenna beam 
and where k = w/c with w ranging over the angular frequencies received by the 
radar receiver. The region determined in this manner is called the data-collection 
manifold. The extent of the set of angles is called the synthetic aperture, and the 
extent of the set of frequencies is called the bandwidth. Typical synthetic apertures 
are on the order of a few degrees, and bandwidths of 27 x 500 x 10° rad/s are not 
uncommon. Figure 2 shows an example of data-collection manifold corresponding 
to turntable geometry; Fig. 3 shows others that correspond to more complex motion. 
Typical SAR data-collection manifolds are two-dimensional manifolds. The larger 
the data-collection manifold at z, the better the resolution at z. 
Examples of ISAR images are shown in Figs. 4 and 5. 


ISAR in the Time Domain 
Fourier transforming (32) into the time-domain results in 


nB(t, On) X // eit 218°l/e+20 (6n)® *¥/2)) Py (ey) |? dew q(y) dy. (34) 


Evaluation of nz at a shifted time results in the simpler expression 


2\x9 . n 
ne (: 422), 2) = // e200 2-¥/0)| Py (ep)? dw q(ydy. (35) 
Cc 
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Fig. 3 The dark surfaces represent some typical data-collection manifolds that are subsets of a 
more complete “data dome” 


With the temporary notation t = —2O(6,,)x-y/c, the w integral on the right side 
of (35) can be written as 


i: 2) Pow)? do = ij 5(s — 1) BU — 5) ds, ais 


where 


A(t —s) = / ei] Py (aw) 2 da. 


With (36), ng can be written 


NB (+44) = [ pus) [ 6(5+ Oy) arava 


~20(6,)z 
ae 


’ 


= 6 «Ral ( 


where 


Rigs, a) = i 5(s — A-y)qv) dy G7) 


is the Radon transform [43,46]. Here ft denotes a unit vector. In other words, the 
Radon transform of q is defined as the integral of g over the plane s = ft -y. 
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Fig. 4 An ISAR image of a Boeing 727 from a 5° aperture [70] 


Fig. 5 On the /eft is an ISAR image of a ship; on the right is an optical image of the same ship 
(Courtesy Naval Research Laboratory) 
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ISAR systems typically use a high-range-resolution (large-bandwidth) wave- 
form, so that B ~ 6 (see section “The Range Profile”). Thus ISAR imaging from 
time-domain data becomes a problem of inverting the Radon transform. 


Synthetic Aperture Radar 


In ISAR, the target rotates and the radar is stationary, whereas in synthetic aperture 
radar (SAR), the target is stationary and the radar moves. (In typical ISAR data- 
collection scenarios, both the radar and the target are actually in motion, and so this 
distinction is somewhat arbitrary.) For most SAR systems [9, 20, 21,29, 35, 60], the 
antenna is pointed toward the earth. For an antenna viewing the earth, an antenna 
beam pattern must be included in the model. For highly directive antennas, often 
simply the antenna “footprint,” which is the illuminated area on the ground, is used. 

For a receiving antenna at the same location as the transmitting antenna, the 
scalar Born model for the received signal is 


Sa(o) = [ 2" 1A(@.2°.VO) dy, G8) 


where A incorporates the geometrical spreading factors |x° — y|~?, transmitted 
waveform, and antenna beam pattern. More details can be found in [15]. 

SAR data-collection systems are usually configured to transmit a series of pulses 
with the nth pulse transmitted at time ¢,,. The antenna position at time ¢,, is denoted 
by y,,. Because the time scale on which the antenna moves is much slower than the 
time scale on which the electromagnetic waves propagate, the time scales separate 
into a slow time, which corresponds to the n of t,, and a fast time t. 

In (38) the antenna position x° is replaced by y,,: 


D(w,n) = F[V](a,s) = [etrl4e@.nyV0) dy, (39) 


where with a slight abuse of notation, the x° in the argument of A has been replaced 
by n. This notation also allows for the possibility that the waveform and antenna 
beam pattern could be different at different points along the flight path. The time- 
domain version of (39) is 


d(t,n) = i: e!lt—21r,-¥/el A (ep, ny) V(y) dy. (40) 


The goal of SAR is to determine V from the data d. 

As in the case of ISAR, assuming that y and A are known, the data depend 
on two variables, so it should be possible to form a two-dimensional image. For 
typical radar frequencies, most of the scattering takes place in a thin layer at the 
surface. The ground reflectivity function V is therefore assumed to be supported on 
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Fig. 6 In spotlight SAR, the radar is trained on a particular location as the radar moves. In 
this figure, the equi-range circles (dotted lines) are formed from the intersection of the radiated 
spherical wave front and the surface of a (flat) earth 


a known surface. For simplicity this surface is assumed to be a flat plane, so that 
V(x) = V(x)6(x3), where x = (x1, x2). 

SAR imaging comes in two basic varieties: spotlight SAR [9,35] and stripmap 
SAR [20, 21, 29, 60]. 


Spotlight SAR 
Spotlight SAR is illustrated in Fig.6. Here, the moving radar system stares at a 
specific location (usually on the ground), so that at each point in the flight path the 
same scene is illuminated from a different direction. When the ground is assumed 
to be a horizontal plane, the iso-range curves are large circles whose centers are 
directly below the antenna at y,,. If the radar antenna is highly directional and the 
antenna footprint is sufficiently far away, then the circular arcs within the footprint 
can be approximated as lines. Consequently, the imaging method is mathematically 
the same as that used in ISAR. 

In particular, the origin of coordinates is taken within the footprint, and the small- 
scene expansion is used, which results in an expression for the matched-filtered 
frequency-domain data: 


D(o,n) = ela / e2n YV(y)A(w,n,y) dy. (41) 


Within the footprint, A is approximated as a product A = A ;(w,n)A2(y). The 
function A; can be taken outside the integral; the function Az can be divided out 
after inverse Fourier transforming. 

As in the ISAR case, the time-domain formulation of spotlight SAR leads to a 
problem of inverting the Radon transform. An example of a spotlight SAR image is 
shown in Fig. 12. 
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Fig. 7 Stripmap SAR acquires data without staring. The radar typically has fixed orientation with 
respect to the flight direction, and the data are acquired as the beam footprint sweeps over the 
ground 


Stripmap SAR 
Stripmap SAR sweeps the radar beam along with the platform without staring at 
a particular location on the ground (Fig. 7). The equi-range curves are still circles, 
but the data no longer depend only on the direction from the antenna to the scene. 
Moreover, because the radar does not stare at the same location, there is no natural 
origin of coordinates for which the small-scene expansion is valid. 

To form a stripmap SAR image, the expression (39) must be inverted without the 
help of the small-scene approximation. One strategy is to use a filtered adjoint of 
the forward map F' defined by Eq. (39). 


The Formal Adjoint of F The adjoint F* is an operator such that 


(LF e\on=(F' hex (42) 


where (-,-) denotes inner product in the appropriate variables. More specifi- 
cally, (42) can be written as 


/ fla) FR)" a) dwas = / (Ft A) @)g" (x) dx. (43) 


Use of (39) in (43) and an interchange of the order of integration leads to 


Fi f(x) = | / eiKI7()—*1 A (ew, 5,x) f(w, 5) da ds. (44) 
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The Imaging Operator Thus, the imaging operator is assumed to be of the form 


I(z) = B[D](@) = // e*lv()-27l O(w, 5,2) D(w, s) daw ds, (45) 


where zr = (z,0) and Q is a filter to be determined below. The time-domain 
version is 


I(z) = Bld|(z) = // eio'Alv() 7/9) O(@, 5,2) dwd(t, 8) ds dt. (46) 


If the filter Q were to be chosen to be identically 1, then, because of (5), the time- 
domain inversion would have the form 


12) = | 6(t — 2|y(s) —zr|/c)d(t, s) ds dt 


= J aeiv) -ari/e.s)as, (47) 


which can be interpreted as follows: at each antenna position s, the data is 
backprojected (smeared out) to all the locations z that are at the correct travel 
time 2|y(s) — zr|/c from the antenna location y(s). Then all the contributions are 
summed coherently (i.e., including the phase). Figure 8 shows the partial sums over 
s as the antenna (white triangle) moves along a straight flight path from bottom to 
top. 

An alternative interpretation is that to form the image at the reconstruction point 
z, all the contributions from the data at all points (t, s) for which t = 2|y(s) —zr|/c 
are coherently summed. 

Note the similarity between (47) and (61): (61) backprojects over lines or planes, 
whereas (47) backprojects over circles. The inversion (46) first applies the filter 0 
and then backprojects. 


Other SAR Algorithms The image formation algorithm discussed here is filtered 
backprojection. This algorithm has many advantages, one of them being great 
flexibility. This algorithm can be used for any antenna beam pattern, for any flight 
path, and for any waveform; a straightforward extension [48] can be used in the case 
when the topography is not flat. 

Nevertheless, there are various other algorithms that can be used in special cases, 
for example, if the flight path is straight, if the antenna beam is narrow, or if a 
chirp waveform is used. Discussions of these algorithms can be found in the many 
excellent radar imaging books such as [9, 20, 29, 35, 61]. 
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Resolution for ISAR and Spotlight SAR 


To determine the resolution of an ISAR image, the relationship between the image 
and the target is analyzed. 

For turntable geometry, (30) is used. The viewing direction is taken to be = 
(1,0, 0), with ég = O(0)x° and k = 2k. Then (32) is proportional to 


Bika) = f esq ay 


= // ek cmtnsnl) f a(y1, ya, ¥8) dy3 dy; dy». (48) 


—— 
q(y1.y2) 


The data depend only on the quantity (1,2) = f ¢(v1.¥2, 3) dy3, which is a 
projection of the target onto the plane orthogonal to the axis of rotation. In other 
words, in the turntable geometry, the radar imaging projection is the projection onto 
the horizontal plane. With the notation y = (y1, y2), so that y = (y, y3), it is clear 
that ég9-y = (Pég)-y, where P : R* —> R? denotes the projection onto the first two 
components of a three-dimensional vector. 

The data-collection manifold Q = {kég : w, < @ < wand |@| < ®} is shown 
in Fig. 2. Then (48) can be written as 


D(k,0) = yo(kes)FIg\ (kes), (49) 


where 7a (ké) denotes the function that is 1 if keg € Q and 0 otherwise. 
The image is formed by taking the two-dimensional inverse Fourier transform 
of (49): 


I(x) = [erie de. ek akao « f er herin [fers Pn gty) aykeak do 
Q 
= if i ell) “k(P08) E ak dO G(y) dy. (50) 
Q 
EE a 
K(x—y) 


The function K is the point-spread function (PSF); it is also called the imaging 
kernel, impulse response, or sometimes ambiguity function. The PSF can be written 
as 


et Bie hee yeh es lEl=kn po aie hie. fs 
K¢x) « ff on tring aka = [ ; / eX KPO dk dé. (51) 
Q |gé|=k; J-—® 
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It can be calculated by writing 
x=r(cosy,sinw) and (Pé,) = (cos¢, sing), (52) 


so that x-(Pé9) = r cos(¢—). The “down-range” direction corresponds to w = 0 
and “cross-range” corresponds to w = 2/2. 


Down-Range Resolution in the Small-Angle Case 
For many radar applications, the target is viewed from only a small range of aspects 
€9; in this case, the small-angle approximations cos¢@ ~ 1 and sing ~ @¢ can be 
used. 

In the down-range direction (w = 0), under the small-angle approximation, (51) 


becomes 
kn p® 7 
K(r,0) © [ k L e" dob dk 
ky - 


20 d 
= 20 ° gir dk = a elk ak 
ky 


ky i 
2d[ub. b 
=== ie 5 sine | (53) 


where b = ks _ k= = 47 B/c, B is the bandwidth in hertz, and ko = (ky + ko) /2 = = 
2a(vyy + v2) = = 2m Vo, where vo is the center frequency in hertz. 

Since ko > Db, the leading order term of (53) is obtained by differentiating the 
exponential: 


a oe 
K(r,0) © bko® elk" sine br, (54) 


yielding peak-to-null down-range resolution 277/b = c/(2B). Here, it is the sinc 
function that governs the resolution. 


Cross-Range Resolution in the Small-Angle Case 

In the cross-range direction (y = 2/2), the approximation cos(@—w) = sing = ¢ 
holds under the small-angle assumption. With this approximation, the computation 
of (51) is 


ko p® i 
K(0,r) © [ k | ed dob dk 
ky —® 


ko _ aikr® -ikr® 
2~@ —e a 
7 k ————__ dk 
ky ikr 


7 os, 1 a 1 
— | e"®p sinc | —br® | — e*0"®p sinc ( —br® 
ir 2 : 


2bko® sinc (5oro) sinc(kgr®). (55) 


II 


II 


II 
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Since ko > b, 


K(0,r) & 2bko® sinc(kor®). (56) 


Thus the peak-to-null cross-range resolution is / (kp®) =c /(4vo®) = Ao/(4®). 
Since the angular aperture is 2®, the cross-range resolution is Ao divided by twice 
the angular aperture. 


Example Figure 9 shows a numerical calculation of K for @ = 12° and two 
different frequency bands: [k;,k2] = [0,300] (i.ec., b = 300 and ko = 150, and 
[k1,k2] = [200, 300]) (i.e., b = 100 and kg = 250). The first case is not relevant 
for most radar systems, which do not transmit frequencies near zero, but is relevant 
for other imaging systems such as X-ray tomography. These results are plotted in 
Fig. 9. 


A 
Supp (K) radon Re (K) radon Cross-section 


—400 -200 
~ 
Supp (K) radon Re (K) radon Cross-section 
t 4 4 4 
—400 -200 


UL4000 


Fig.9 From left to right: the data-collection manifold, the real part of K, cross sections 
(horizontal is rapidly oscillating; vertical is slowly oscillating) through the real part of K for the 
two cases. Down range is horizontal (Reprinted with permission from [45]) 
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5 Numerical Methods 


ISAR and Spotlight SAR Algorithms 


The Polar Format Algorithm (PFA) For narrow-aperture, turntable-geometry 
data, such as shown in Fig. 2, the polar format algorithm (PFA) is commonly used. 
The PFA consists of the following steps, applied to frequency-domain data: 


1. Interpolate from a polar grid to a rectangular grid (see Fig. 10.) 
2. Use the two-dimensional discrete (inverse) Fourier transform to form an image 


of q. 


Alternatively, algorithms for computing the Fourier transform directly from a 
nonuniform grid can be used [33, 46, 57]. 


Inversion by Filtered Backprojection For the n-dimensional Radon transform, 
one of the many inversion formulas [43, 44] is 


1 


= ia (R[f]). (57) 


# 


where Z!~" is the Riesz operator (filter) 
Tf =F" [vl F fs] (58) 


operating on the s variable, and the operator R* is the formal adjoint of R. (Here 
the term “formal” means that the convergence of the integrals is not considered; the 


Polar data grid 


Rectangular 
interpolating grid 


Fig. 10 This illustrates the 
process of interpolating from o — Measured data 


a polar grid to a rectangular e — Interpolated data 
grid 
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identities are applied only to functions that decay sufficiently rapidly, so that the 
integrals converge.) The adjoint is defined by the relation 


(Rh) >= (fRih)s, (59) 
where 
(Rh), ff R.A" .i asa (60) 
and 
(ERI = f SOIR) ae. 


Using (37) in (60) and interchanging the order of integration show that the adjoint 
Ri operates on h(s, sw) via 


(R'm) = fnew a. fap. 61) 


Here R* integrates over the part of h corresponding to all planes (n = 3) or 
lines (n = 2) through x. When R* operates on Radon data, it has the physical 
interpretation of backprojection. For example, in the case where / represents Radon 
data from a point-like target, for a fixed direction jt, the quantity h(x-j, jt), as a 
function of x, is constant along each plane (or line ifn = 2)x- ™& = constant. Thus, 
at each ji, the function h(x - jv, f£) can be thought of as an image in which the data 
for direction u is backprojected (smeared) onto all points x that could have produced 
the data for that direction. The integral in (61) then sums the contributions from all 
the possible directions. (See Fig. 11.) The inversion formula (57) is thus a filtered 
backprojection formula. Fast backprojection algorithms have been developed by a 
number of authors (e.g., [24, 76]). 


Range Alignment 


ISAR imaging relies on target/radar relative motion. An assumption made through- 
out is that the target moves as a rigid body — an assumption that ignores the flexing of 
aircraft lift and control surfaces or the motion of vehicle treads. Moreover, arbitrary 
rigid-body motion can always be separated into a rotation about the body’s center of 
mass and a translation of that center of mass. Backprojection shows how the rotation 
part of the relative radar/target motion can be used to reconstruct a two-dimensional 
image of the target in ISAR and spotlight SAR. But, usually while the target is 
rotating and the radar system is collecting data, the target will also be translating, 
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Fig. 11 This figure illustrates the process of backprojection. The range profiles (inset) suggest 
the time delays that would result from an interrogating radar pulse incident from the indicated 
direction. Note that scatterers that lie at the same range from one view do not lie at the same range 
from other views 


and this has not been accounted for in the imaging algorithm. In Eqs. (27) and (28), 
for example, R = |xo| was implicitly set to a constant (Fig. 12). 

Typically, the radar data are preprocessed to subtract out the effects of target 
translation before the imaging step is performed. Under the start-stop approxima- 
tion, the range profile data 7, (t, @,) is approximately a shifted version of the range 
profile (see section “The Range Profile”) at the previous pulse. Thus g(t, 0,41) © 
ne(t + Aty,9,), where At, is a range offset that is determined by target motion 
between pulses. 

The collected range profiles can be shifted to a common origin if Az, can be 
determined for each 6,,. One method to accomplish this is to assume that one of the 
peaks in each of the range profiles (e.g., the strongest peak) is always due to the 
same target feature and so provides a convenient origin. This correction method is 
known as “range alignment” and must be very accurate in order to correct the offset 
error within a fraction of a wavelength. (Note that the wavelength in question is 
that of the signal output by the correlation receiver and not the wavelength of the 
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y (m) 


Fig. 12 A radar image from a circular flight path, together with an optical image of the same 
scene. The bright ring in the top half of the radar image is a “top-hat” calibration target used to 
focus the image (Courtesy US Air Force Sensors Directorate) 


transmitted waveform. In HRR systems, however, this wavelength can still be quite 
small.) Typically, 73 (t + At,, 0,) is correlated with nx, (t, 6,41), and the correlation 
maximum is taken to indicate the size of the shift Azt,. This idea is illustrated in 
Fig. 13 which displays a collection of properly aligned range profiles. 

When the scattering center used for range alignment is not a single point but, 
rather, several closely spaced and unresolved scattering centers, then additional con- 
structive and destructive interference effects can cause the range profile alignment 
feature — assumed to be due to a single well-localized scatterer — to vary rapidly 
across the synthetic aperture (i.e., such scattering centers are said to “scintillate’’). 
For very complex and scintillating targets, other alignment methods are used: for 
example, if the target is assumed to move along a “smooth” path, then estimates 
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Fig. 13 Range alignment preprocessing in synthetic aperture imaging. The effects of target 
translation must be removed before backprojection can be applied 


of its range, range rate, range acceleration, and range jerk (time derivative of 
acceleration) can be used to express target range as a polynomial in time 


. diez 1 ... 
R(6n) = R(O) + ROn + 56, ~ Ron (62) 
In terms of this polynomial, 


3 R(6,) — RO) _ , RO, + 4 RO? + 2 RO? 
c Cc 


Atn , (63) 


where R, R, and R are radar measurables. 

Of course, the need for range alignment preprocessing is not limited to ISAR 
imaging; similar motion compensation techniques are needed in SAR as well 
(Fig. 14). 


6 Open Problems 


In the decades since the invention of synthetic aperture radar imaging, there has 
been much progress, but many open problems still remain. And most of these open 
problems are mathematical in nature. 

As outlined at the beginning of Sect.3, SAR imaging is based on specific 
assumptions, which in practice may not be satisfied. When they are not satisfied, 
artifacts appear in the image. 
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Motion compensated 


Three-meter resolution Gy 


Fig. 14 The effect of motion compensation in a Ku-band image (Courtesy Sandia National 
Laboratories) 


Problems Related to Unmodeled Motion 


SAR image formation algorithms assume the scene to be stationary. Motion in the 
scene gives rise to mispositioning or streaking (see Figs. 15 and 16). This effect is 
analyzed in [28]. 

However, it is of great interest to use radar to identify moving objects; systems 
that can do this are called moving target indicator (MTI) systems or ground moving 
target indicator (GMTI) systems. 


1. How can artifacts associated with targets that move during data collection [54] be 
mitigated? Moving targets cause Doppler shifts and also present different aspects 
to the radar [14]. An approach for exploiting unknown motion is given in [65]. 

2. Both SAR and ISAR are based on known relative motion between the target and 
sensor, for example, the assumption that the target behaves as a rigid body. When 
this is not the case, the images are blurred or uninterpretable. Better methods for 
finding the relative motion between the target and sensor are also needed [6,65]. 
Better algorithms are needed for determining the antenna position from the radar 
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Fig. 15 A Ku-band image showing streaking due to objects moving in the scene (Courtesy Sandia 
National Laboratories and SPIE) 


data itself. Such methods include autofocus algorithms [35,40], some of which 
use a criterion such as image contrast to focus the image. 

3. When the target motion is complex (pitching, rolling, and yawing), it may be 
possible to form a three-dimensional image; fast, accurate methods for doing 
this are needed [65]. How can moving objects be simultaneously tracked [58] 
and imaged? 


Problems Related to Unmodeled Scattering Physics 


1. How can images be formed without the Born approximation? The Born approx- 
imation leaves out many physical effects, including not only multiple scattering 
and creeping waves but also shadowing, obscuration, and polarization changes. 
But without the Born approximation (or the Kirchhoff approximation, which is 
similar), the imaging problem is nonlinear. In particular, how can images be 
formed in the presence of multiple scattering? (See [6, 13,31, 49, 73].) Artifacts 
due to the Born approximation can be seen in Fig.4, where the vertical streaks 
near the tail are due to multiple scattering in the engine inlets. Can multiple 
scattering be exploited [13,39] to improve resolution? 
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Fig. 16 A 4-in. resolution of SAR image from Sandia National Laboratories. Only certain parts 
of the airplanes reflect radar energy. The inset is an optical image of the airplanes (Courtesy Sandia 
National Laboratories) 


2. Scattering models need to be developed that include as much of the physics as 
possible but are still simple enough for use in the inverse problem. An example 
of a simple model that includes relevant physics is [56]. 

3. How can polarization information [4, 17, 18,55,71] be exploited? This problem 
is closely connected to the issue of multiple scattering: the usual linear models 
predict no change in the polarization of the backscattered electric field. Conse- 
quently linear imaging methods cannot provide information about how scatterers 
change the polarization of the interrogating field. A paper that may be useful here 
is [68]. 

4. How can prior knowledge about the scene be incorporated in order to improve 
resolution? There is interest in going beyond simple aperture/bandwidth-defined 
resolution [45,66]. One approach that has been suggested is to apply compressive 
sensing ideas [1, 10,42] to SAR. 

5. How can information in the radar shadow be exploited? In many cases it is 
easier to identify an object from its shadow than from its direct-scattering image. 
(See Fig. 17.) A backprojection method for reconstructing an object’s three- 
dimensional shape from its shadows obtained at different viewing angles is 
proposed in [23]. What determines the resolution of this reconstruction? 


Synthetic Aperture Radar Imaging 795 


Fig. 17 A 4-in. resolution image from Sandia National Laboratories. Note the shadows of the 
historical airplane, helicopter, and trees (Courtesy of Sandia National Laboratories) 


New Applications of Radar Imaging 


1. Can radar systems be used to identify individuals by their gestures or gait? Time- 
frequency analysis of radar signals gives rise to micro-Doppler time-frequency 
images [11], in which the motion of arms and legs can be identified. 

2. How can radar be used to form images of urban areas? It is difficult to form SAR 
images of urban areas, because in cities the waves undergo complicated multipath 
scattering. Areas behind buildings lie in the radar shadows, and images of tall 
buildings can obscure other features of interest. In addition, urban areas tend to 
be sources of electromagnetic radiation that can interfere with the radiation used 
for imaging. 

One approach that is being explored is to use a persistent or staring radar sys- 
tem [27] that would fly in circles [61] around a city of interest (see, e.g., Fig. 12). 
Thus, the radar would eventually illuminate most of the areas that would be shad- 
owed when viewed from a single direction. However, this approach has the added 
difficulty that the same object will look different when viewed from different 
directions. How can the data from a staring radar system be used to obtain the 
maximum amount of information about the (potentially changing) scene? 
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3. If sensors are flown on unoccupied aerial vehicles (UAVs), where should these 
UAVs fly? The notion of swarms of UAVs [2] gives rise not only to challenging 
problems in control theory but also to challenging imaging problems. 

4. Many of these problems motivate a variety of more theoretical open problems 
such as the question of whether backscattered data uniquely determines a 
penetrable object or a non-convex surface [63,72]. There is a close connection 
between radar imaging and the theory of Fourier integral operators [48]. How 
can this theory be extended to the case of dispersive media and to nonlinear 
operators? Is it possible to develop a theory of the information content [37, 52] 
of an imaging system? 


7 Conclusion 
Radar imaging is a mathematically rich field with many interesting open problems. 
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"Abstract 


We define tomography as the process of producing an image of a distribution (of 
some physical property) from estimates of its line integrals along a finite number 
of lines of known locations. We touch upon the computational and mathematical 
procedures underlying the data collection, image reconstruction, and image 
display in the practice of tomography. The emphasis is on reconstruction 
methods, especially the so-called series expansion reconstruction algorithms. 

We illustrate the use of tomography (including three-dimensional displays 
based on reconstructions) both in electron microscopy and in X-ray computerized 
tomography (CT), but concentrate on the latter. This is followed by a classifica- 
tion and discussion of reconstruction algorithms. In particular, we discuss how to 
evaluate and compare the practical efficacy of such algorithms. 


G.T. Herman (1) 

Department of Computer Science, The Graduate Center of the City University of New York, 
New York, NY, USA 

e-mail: gabortherman @ yahoo.com 


© Springer Science+Business Media New York 2015 801 
O. Scherzer (ed.), Handbook of Mathematical Methods in Imaging, 
DOI 10.1007/978-1-4939-0790-8_16 


802 G.T. Herman 


1 Introduction 


To get the flavor of tomography in general, we first discuss a special case: X- 
ray computerized tomography (CT) for reconstructing the distribution within a 
transverse section of the human body of a physical parameter (the “relative linear 
attenuation at energy e” whose value at the point (x, y) in the section is denoted 
by z(x, y)) from multiple X-ray projections. A typical method by which data 
are collected for transverse section imaging in CT is indicated in Fig. 1. A large 
number of measurements are taken. Each of these measurements is related to an 
X-ray source position combined with an X-ray detector position, and from the 
measurements we can (based on physical principles) estimate the line integral of 
[tz along the line between the source and the detector. The mathematical problem 
is: given a large number of such projections, reconstruct the image juz(x, y). 

A chapter such as this can only cover a small part of what is known about 
tomography. A much extended treatment in the same spirit as this chapter is given in 
[23]. For additional information on mathematical matters related to CT, the reader 
may consult the books [7, 17,24, 29, 50]. In particular, because of the mathematical 
orientation of this handbook, we will not get into the details of the how the line 
integrals are estimated from the measurements. (Such details can be found in [23]. 
They are quite complicated: in addition to the actual measurement with the patient 
in the scanner a calibration measurement needs to be taken, both of these need to be 
normalized by the reference detector indicated in Fig. 1, correction has to be made 
for the beam hardening that occurs due to the X-ray beam being polychromatic 
rather than consisting of photons at the desired energy é, etc.) 


2 Background 


The problem of image reconstruction from projections has arisen independently 
in a large number of scientific fields. A most-important version of the problem 
in medicine is CT; it has revolutionized diagnostic radiology over the past four 
decades. The 1979 Nobel prize in physiology and medicine was awarded to Allan 
M. Cormack and Godfrey N. Hounsfield for the development of X-ray computerized 
tomography [9, 31]. The 1982 Nobel prize in chemistry was awarded to Aaron 
Klug, one of the pioneers in the use of reconstruction from electron microscopic 
projections for the purpose of elucidation of biologically important molecular 
complexes [11, 13]. The 2003 Nobel prize in physiology and medicine was awarded 
to Paul C. Lauterbur and Peter Mansfield for their discoveries concerning magnetic 
resonance imaging, which also included the use of image reconstruction from 
projections methods [38]. 

In some sense this problem was solved in 1917 by Johann Radon [52]. Let @ 
denote the distance of the line L from the origin, let 6 denote the angle made with 
the x axis by the perpendicular drawn from the origin to L (see Fig. 1), and let 
m(£, 6) denote the integral of jzz along the line L. Radon proved that 
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Fig. 1 Data collection for CT (reproduced from [23]) 
1 CO 1 Qn 
be(x, vy) = -= tim f - | mi(xcos@ + ysin@ + q,0)dé dq, (1) 
one £0 J, q Jo 


where m,(£,@) denotes the partial derivative of m(£,6) with respect to £. The 
implication of this formula is clear: the distribution of the relative linear attenuation 
in an infinitely thin slice is uniquely determined by the set of ail its line integrals. 
However: 


(a) Radon’s formula determines an image from all its line integrals. In CT we have 
only a finite set of measurements; even if they were exactly integrals along lines, 
a finite number of them would not be enough to determine the image uniquely, 
or even accurately. Based on the finiteness of the data one can produce objects 
for which the reconstructions will be very inaccurate (Section 15.4 of [23]). 

(b) The measurements in computed tomography can only be used to estimate the 
line integrals. Inaccuracies in these estimates are due to the width of the X-ray 
beam, scatter, hardening of the beam, photon statistics, detector inaccuracies, 
etc. Radon’s inversion formula is sensitive to these inaccuracies. 

(c) Radon gave a mathematical formula; we need an efficient algorithm to evaluate 
it. This is not necessarily trivial to obtain. There has been a very great deal of 
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ee 


Fig. 2. Engineering rendering of a 2008 CT scanner (provided by GE Healthcare) 


activity to find algorithms that are fast when implemented on a computer and yet 
produce acceptable reconstructions in spite of the finite and inaccurate nature of 
the data. This chapter concentrates on this topic. 


3 Mathematical Modeling and Analysis 


The mathematical model for CT is illustrated in Fig. 1. An engineering realization of 
this model is shown in Fig. 2. The tube contains a single X-ray source, the detector 
unit contains an array of X-ray detectors. Suppose for the moment that the X-ray 
Tube and Collimator on the one side and the Data Acquisition/Detector Unit on the 
other side are stationary, and the patient on the table is moved between them at a 
steady rate. By shooting a fan beam of X-rays through the patient at frequent regular 
intervals and detecting them on the other side, we can build up a two-dimensional 
X-ray projection of the patient that is very similar in appearance to the image that 
is traditionally captured on an X-ray film. Such a projection is shown in Fig. 3a. 
The brightness at a point is indicative of the total attenuation of the X-rays from the 
source to the detector. This mode of operation is not CT, it is just an alternative way 
of taking X-ray images. In the CT mode, the patient is kept stationary, but the tube 
and the detector unit rotate (together) around the patient. The fan beam of X-rays 
from the source to the detector determines a slice in the patient’s body. The location 
of such a slice is shown by the horizontal line in Fig. 3a. 

Data are collected for a number of fixed positions of the source and detector; 
these are referred to as views. For each view, we have a reading by each of 
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Fig. 3 (a) Digitally radiograph with /ine marking the location of the cross section for which the 
following images were obtained. (b) Sinogram of the projection data. (c) Reconstruction from the 
projection data (images were obtained using a Siemens Sensation CT scanner by R. Fahrig and J. 
Starman at Stanford University) 


the detectors. All the detector readings for all the views can be represented as a 
sinogram, shown in Fig.3b. The intensities in the sinogram are proportional to 
the line integrals of the X-ray attenuation coefficient between the corresponding 
source and detector positions. From these line integrals, a two-dimensional image 
of the X-ray attenuation coefficient distribution in the slice of the body can be 
produced by the techniques of image reconstruction. Such an image is shown in 
Fig. 3c. Inasmuch as different tissues have different X-ray attenuation coefficients, 
boundaries of organs can be delineated and healthy tissue can be distinguished from 
tumors. In this way CT produces cross-sectional slices of the human body without 
surgical intervention. 

We can use the reconstructions of a series of parallel transverse sections to 
discover and display the precise shape of selected organs; see Fig. 4. Such displays 
are obtained by further computer processing of the reconstructed cross sections [59]. 

As a second illustration of the many applications of tomography (for a more 
complete coverage see Section 1.1 of [23]), we note that three-dimensional recon- 
struction of nano-scale objects (such as biological macromolecules) can be accom- 
plished using data recorded with a transmission electron microscope (see Fig.5) 
that produces electron micrographs, such as the one illustrated in Fig. 6, in which 
the grayness at each point is indicative of a line integral of a physical property of 
the object being imaged. From multiple electron micrographs one can recover the 
structure of the object that is being imaged; see Fig. 7. 

What we have just illustrated in our electron microscopy example is a reconstruc- 
tion of a three-dimensional object from two-dimensional projections; as opposed to 
what is shown in Fig. 1, which describes the collection of data for the reconstruction 
of a two-dimensional object. In fact, recently developed CT scanners are not like 
that, they collect a series of two-dimensional projections of the three-dimensional 
object to be reconstructed. 

Helical CT (also referred to as spiral CT) first started around 1990 [10, 34] 
and has become standard for medical diagnostic X-ray CT. Typical state-of-the- 
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Fig. 4 Three-dimensional displays of bone structures of patients produced during 1986-1988 by 
software developed in the author’s research group at the University of Pennsylvania for the General 
Electric Company. (a) Facial bones of an accident victim prior to operation. (b) The same patient at 
the time of a 1-year postoperative follow-up. (c) A tibial fracture. (d) A pelvic fracture (reproduced 
from [23]) 


art versions of such systems have a single X-ray source and multiple detectors in 
a two-dimensional array. The main innovation over previously used technologies 
is the presence of two independent motions: while the source and detectors rotate 
around the patient, the table on which the patient lies is continuously moved between 
them (typically orthogonally to the plane of rotation), see Fig. 8. Thus, the trajectory 
of the source relative to the patient is a helix (hence the name “helical CT’). 
Helical CT allows rapid imaging as compared with the previous commercially 
viable approaches, which has potentially many advantages. One example is when 
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Fig. 5 Schematic drawing of a transmission electron microscope (illustration provided by C. San 
Martin of the Centro Nacional de Biotecnologia, Spain) 


we wish to image a long blood vessel that is made visible to X-rays by the injection 
of some contrast material: helical CT may very well allow us to image the whole 
vessel before the contrast from a single injection washes out and this may not be 
possible by the slower scanning modes. We point out that the CT scanner illustrated 
in Fig. 2 is in fact modern helical CT scanner. 

For the sake of not over-complicating our discussion, in this chapter we restrict 
our attention (except where it is explicitly stated otherwise) to the problem of 
reconstructing two-dimensional objects from one-dimensional projections, rather 
than to what is done by modern helical cone-beam scanning (as in Fig. 8) and 
volumetric reconstruction. Schematically, the method of our data collection is shown 
in Fig.9. The source and the detector strip are on either side of the object to be 
reconstructed and they move in unison around a common center of rotation denoted 
by O in Fig. 9. The data collection takes place in M distinct steps. The source and 
detector strip are rotated between two steps of the data collection by a small angle, 
but are assumed to be stationary while the measurement is taken. The M distinct 
positions of the source during the M steps of the data collection are indicated by 
the points So,..., Sy—1 in Fig.9. In simulating this geometry of data collection, 
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Fig. 6 Part of an electron 
micrograph containing 
projections of multiple copies 
of the human adenovirus type 
5 (illustration provided by C. 
San Martin of the Centro 
Nacional de Biotecnologia, 
Spain) 


Fig. 7 Top: Reconstructed values, from electron microscopic data such as in Fig. 6, of the human 
adenovirus type 5 in three mutually orthogonal slices through the center of the reconstruction. 
Bottom: Computer graphic display of the surface of the virus based on the three-dimensional 
reconstruction (illustration provided by C. San Martin of the Centro Nacional de Biotecnologia, 
Spain) 
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Fig. 8 Helical (also known as spiral) CT (illustration provided by G. Wang of the Virginia 
Polytechnic Institute & State University) 
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Fig. 9 Schematic of a standard method of data collection (divergent beam). This is consistent with 
the data collection mode for CT that is shown in Fig. | (reproduced from [23]) 


we assume that the source is a point source. The detector strip consists of 2N + 1 
detectors, spaced equally on an arc whose center is the source position. The line 
from the source to the center of rotation goes through the center of the central 
detector. (This description is that of the geometry that is assumed in much of what 
follows and it does not exactly match the data collection by any actual CT scanner. 
In particular, in real CT scanners the central ray usually does not go through the 
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middle of the central detector, as a 1/4 detector offset is quite common.) The object 
to be reconstructed is a picture such that its picture region (i.e., a region outside of 
which the values assigned to the picture are zero) is enclosed by the broken circle 
shown in Fig.9. We assume that the origin of the coordinate system (with respect 
to which the picture values j1z(x, y) are defined) is the center of rotation, O, of the 
apparatus. 

Until now we have used juz(x, y) to denote the relative linear attenuation at the 
point (x, y), where (x, y) was in reference to a rectangular coordinate system, see 
Fig. |. However, it is often more convenient to use polar coordinates. We use the 
phrase a function of two polar variables to describe a function f whose values 
f(r, @) represent the value of some physical parameter (such as the relative linear 
attenuation) at the geometrical point whose polar coordinates are (7, ¢). 

We define the Radon transform & f of a function f of two polar variables as 
follows: for any real number pairs (£, 0), 


aN. = [ f(VEFR o+ tan 'G/O) dz, ite 40. 
i (2) 
(Af 0.0)= ff O+x/2dz 


Observing Fig. 1, we see that [% f] (€, 0) is the line integral of f along the line 
L. (Note that the dummy variable z in (2) does not exactly match the variable z as 
indicated in Fig. 1. In (2) z = 0 corresponds to the point where the perpendicular 
dropped on L from the origin meets L.) 

In tomography we may assume that a picture function has bounded support; i.e., 
that there exists a real number £, such that f(r,@) = Oifr > E.(E can be chosen 
as the radius of the broken circle in Fig. 9, which should enclose the square-shaped 
reconstruction region in Fig. 1.) For such a function, [% f] (€,0) = 0if 2 > E. 

The input data to a reconstruction algorithm are estimates (based on physical 
measurements) of the values of [% f] (€, 9) for a finite number of pairs (¢, 0); its 
output is an estimate, in some sense, of f. Suppose that estimates of [# f] (€, @) 
are known for J pairs: (€;,01),..., (€7, 97). For 1 <i < I, we define &; f by 


Ki f =|#f\ (Gi, 6). (3) 
In what follows we use y; to denote the available estimate of &; f and we use y to 
denote the /-dimensional vector whose ith componentis y;. We refer to the vector y 
as the measurement vector. When designing a reconstruction algorithm we assume 
that the method of data collection, and hence the set {(£1, 01) ,..., (€r, 97)}, is fixed 
and known. The reconstruction problem is 


given the data y, estimate the picture f. 


We shall usually use f* to denote the estimate of the picture f. 
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In the mathematical idealization of the reconstruction problem, what we are 
looking for is an operator #~!, which is an inverse of & in the sense that, for 
any picture function f, 2 '&f is f (iec., Z! associates with the function Z f 
the function f). Just as (2) describes how the value of # f is defined at any real 
number pair (£, 6) based on the values f assumes at points in its domain, we need 
a formula that for functions p of two real variables defines Z~! p at points (r, #). 
Such a formula is 


1 nk 1 
[Zp]. 6) = =a | if. rob apy aarie ade dé, (4) 


where p)(£,6) denotes the partial derivative of p(€, 0) with respect to €; it is of 
interest to compare this formula with (1). That the Z~! defined in this fashion is 
indeed the inverse of Z is proven, for example, in Section 15.3 of [23]. 

A major category of algorithms for image reconstruction calculate {* based 
on (4), or on alternative expressions for the inverse Radon transform #~!. We refer 
to this category as transform methods. While (4) provides an exact mathematical 
inverse, in practice it needs to be evaluated based on finite and imperfect data using 
the not unlimited capabilities of computers. The essence of any transform method 
is a numerical procedure (i.e., one that can be implemented on a digital computer), 
which estimates the value of a double integral, such as the one that appears on the 
right-hand side of (4), from given values of y; = p(€;,9;), 1 <i < I. A very 
widely used example of transform methods is the so-called filtered backprojection 
(FBP) algorithm. The reason for this name can be understood by looking at the right- 
hand side of (4): the inner integral is essentially a filtering of the projection data for 
a fixed @ and the outer integral backprojects the filtered data into the reconstruction 
region. However, the implementational details for the divergent beam data collection 
specified in Fig. 9 are less than obvious, the solution outlined below is based on [28]. 


The data collection geometry we deal with is also described in Fig. 10. The X-ray 
source is always on a circle of radius D around the origin. The detector strip is an arc 
centered at the source. Each line can be considered as one of a set of divergent lines 
(o, B), where 6 determines the source position and o determines which of the lines 
diverging from this source position we are considering. This is an alternative way 
of specifying lines to the (£, @) notation used previously (in particular in Fig. 1). Of 
course, each (0, 6) line is also an (£, @) line, for some values of £ and 6 that depend 
ono and B. We use g(o, B) to denote the line integral of f along the line (o, 8). 
Clearly, 


g(o,B) = (#f](D sino, B +o). (5) 


As shown in Fig. 9, we assume that projections are taken for M equally spaced 
values of 6 with angular spacing A, and that for each view the projected values 
are sampled at 2N + 1 equally spaced angles with angular spacing 4. Thus g is 
known at points (nA, mA), -N <n < N,0 <m < M-—1,and MA = 27. 
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Fig. 10 Geometry of divergent beam data collection. Every one of the diverging lines is 
determined by two parameters f and o. Let O be the origin and S be the position of the source, 
which always lies on a circle of radius D around O. Then 6 + 2/2 is the angle the line OS makes 
with the baseline B and o is the angle the divergent line makes with SO. The divergent line is also 
one of a set of parallel lines. As such it is determined by the parameters £ and 6. Let P be the 
point at which the divergent line meets the line through O that is perpendicular to it. Then ¢ is the 
distance from O to P and 6 is the angle that OP makes with the baseline (reproduced from [22], 
Copyright 1981) 


Even though the projection data consist of estimates (based on measurements) 
of g(nA,mA), we use the same notation g(nA,mA) for these estimates. The 
numerical implementation of the FBP method for divergent beams is carried out 
in two stages. 

First we define, for —N <n’ < N, 


N 
ec(n’'A,mA) =A se cos(nA)g(nd, mA)g™ ((n! — n)a) 
n=—N 
N 
+A cos(n’A) »- g(na,mA)q® ((n' —n)A). (6) 


n=—N 


The functions g‘) and q® determine the nature of the “filtering” in the filtered 
backprojection method. They are not arbitrary, but there are many possible choices 
for them, for a detailed discussion see Chapter 10 of [23]. Note that the first sum 
in (6) is a discrete convolution of g“ and the projection data weighted by a cosine 
function, and the second sum is a discrete convolution of g® and the projection 
data. 
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Second we specify our reconstruction by 


M-1 
: DA 1 

ao = res a wee (o’,mA) ; (7) 

where 
o’ = tan! nee) ‘ <2 age", (8) 

D+rsin(mA — @¢) 2 2 
and 
. \1/2 

W= (¢ cos(mA — $))? + (D +r sin(mA — 4)) ) , WO. (9) 


The meanings of o’ and W are that when the source is at angle mA, the line that 
goes through (r,@) is (o’,mA) and the distance between the source and (r, ¢) 
is W. Implementation of (7) involves interpolation for approximating g.(o’, mA) 
from values of g.(n'A,mA). The nature of such an interpolation is discussed in 
some detail in Section 8.5 of [23]. Note that (7) can be described as a “weighted 
backprojection.” Given a point (r, @) and a source position mA, the line (o’, mA) 
is exactly the line from the source position mA through the point (r,@). The 
contribution of the convolved ray sum g,(o’, mA) to the value of f* at points (r, d) 
that the line goes through is inversely proportional to the square of the distance of 
the point (r, @) from the source position mA. 

In this chapter we concentrate on the other major category of reconstruction 
algorithms, the so-called series expansion methods. In transform methods the 
techniques of mathematical analysis are used to find an inverse of the Radon 
transform. The inverse transform is described in terms of operators on functions 
defined over the whole continuum of real numbers. For implementation of the 
inverse Radon transform on a computer we have to replace these continuous 
operators by discrete ones that operate on functions whose values are known only for 
finitely many values of their arguments. This is done at the very end of the derivation 
of the reconstruction method. The series expansion approach is basically different. 
The problem itself is discretized at the very beginning: estimating the function is 
translated into finding a finite set of numbers. This is done as follows. 

For any specified picture region, we fix a set of J basis functions {b,,..., by}. 
These ought to be chosen so that, for any picture f with the specified picture 
region that we may wish to reconstruct, there exists a linear combination of the 
basis functions that we consider an adequate approximation to f. 

An example of such an approach is the 7 x n digitization in which we cover the 
picture region by ann xn array of identical small squares, called pixels. In this case 
J =n’. We number the pixels from 1 to J, and define 
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1, if (r, d)is inside the jth pixel, 
bd) =} 4 eee vee (10) 
, otherwise. 
Then the n x n digitization of the picture f is the picture - defined by 
. J 
f(,6) = > xjbi(9), (11) 


j=l 


where x; is the average value of f inside the jth pixel. A shorthand notation we 
use for equations of this type is FA = yi x;b;. 

Another (and usually preferable) way of choosing the basis functions is the 
following. Generalized Kaiser—Bessel window functions, which are also known by 
the simpler name blobs, form a large family of functions that can be defined in a 
Euclidean space of any dimension [40]. Here we restrict ourselves to a subfamily in 
the two-dimensional plane, whose elements have the form 


v2 r\2 : 
Cons (= (6)') f( i=). Osea, iy: 


0, otherwise, 


Da.pe8 (r, ¢) — 


where J; denotes the modified Bessel function of the first kind of order k, a stands 
for the nonnegative radius of the blob, and @ is a nonnegative real number that 
controls the shape of the blob. The multiplying constant C,,5 is defined below. 
Note that such a blob is circularly symmetric, since its value does not depend on @. 
It has the value zero for all r > a and its first derivatives are continuous everywhere. 
The “smoothness” of blobs can be controlled by the choice of the parameters a, a 
and 6, they can be made very smooth indeed as shown in Fig. 11. 

For now let us consider the parameters a, a and 6, and hence the function by «.s, 
to be fixed. This fixed function gives rise to a set of J basis functions {b),...,b;} 
as follows. We define a set G = {g1,...,g 7} of grid points in the picture region. 
Then, for 1 < j < J, 5; is obtained from bz.,3 by shifting it in the plane so that its 
center is moved from the origin to g;. This definition leaves a great deal of freedom 
in the selection of G, but it was found in practice advisable that it should consist of 
those points of a set (in rectangular coordinates) 


of (28) 


that are also in the picture region. Here 6 has to be a positive real number and G3 
is referred to as the hexagonal grid with sampling distance 5. Having fixed 6, we 
complete the definition in (12) by 


m and n are integers and m + n is even (13) 
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Fig. 11 (a) A 243 x 243 digitization of a blob. (b) Its values on the central row (reproduced from 
[23]) 


J382a 


Cua = ——> ~~: 
ad 4a? 13(a) 


(14) 


Pixel-based basis functions (10) have a unit value inside the pixels and zero 
outside. Blobs, on the other hand, have a bell-shaped profile that tapers smoothly 
in the radial direction from a high value at the center to the value 0 at the edge of 
their supports (i.e., at r = a in (12)); see Fig. 11. The smoothness of blobs suggests 
that reconstructions of the form (11) are likely to be resistant to noise in the data. 
This has been shown to be particularly useful in fields in which the projection data 
are noisy, such as positron emission tomography and electron microscopy. 

For blobs to achieve their full potential, the selection of the parameters a, a, 
and 6 is important. When they are properly chosen [46], one can approximate 
homogeneous regions very well, in spite of the bell-shaped profile of the individual 
blobs. This is illustrated in Fig. 12b, in which a bone cross section shown in Fig. 12a 
is approximated by a linear combination of blob basis functions with the parameters 
a = 0.1551, a = 11.2829, and 6 = 0.0868. There are some inaccuracies very near 
the sharp edges, but the interior of the bone is approximated with great accuracy. 
However, if we change the parameters ever so slightly toa = 0.16, a = 11.28, and 
6 = 0.09, then the best approximation that can be obtained by a linear combination 
of blob basis functions is shown in Fig. 12c, which is clearly inferior. 

Irrespective of how the basis functions have been chosen, any picture 7 that 
can be represented as a linear combination of the basis functions 5; is uniquely 
determined by the choice of the coefficients x;, 1 < j < J, in the formula (11). We 
use x to denote the vector whose jth component is x; and refer to x as the image 
vector. 

It is easy to see that, under some mild mathematical assumptions, 
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Fig. 12 (a) A 243 x 243 digitization of a bone cross section. (b) Its approximation with default 
blob parameters and (c) with slightly different parameters. The display window is very narrow for 
better indication of errors (reproduced from [23]) 


J 
Bi f ~ Bf => xj Bib, (15) 


j=l 


for 1 <i < J. Since the b; are user-defined, usually the 4b; can be easily 
calculated by analytical means. For example, in the case when the 5; are defined 
by (10), &6; is just the length of intersection with the jth pixel of the line of the 
ith position of the source-detector pair. We use r;,; to denote our calculated value 
of %;b;. Hence, 


Vij ~ &;b;. (16) 
Recall that y; denotes the physically obtained estimate of %; f Combining this 
with (15) and (16), we get that, for 1 <i < J, 

J 

K2 > pe (17) 
j=l 

Let R denote the matrix whose (i, 7 )th element is 7;,;. We refer to this matrix 

as the projection matrix. Let e be the J-dimensional column vector whose ith 


component, e;, is the difference between the left- and right-hand sides of (17). We 
refer to this as the error vector. Then (17) can be rewritten as 


y= Rxt+e. (18) 


The series expansion approach leads us to the following discrete reconstruction 
problem: based on (18), 


given the data y, estimate the image vector x. 
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If the estimate that we find as our solution to the discrete reconstruction problem is 
the vector x*, then the estimate {* to the picture to be reconstructed is given by 


J 
ft =o x%by. (19) 


j=l 


In (18), the vector e is unknown. The simple approach of trying to solve (18) by 
first assuming that e is the zero vector is dangerous: y = Rx may have no solutions, 
or it may have many solutions, possibly none of which is any good for the practical 
problem at hand. Some criteria have to be developed, indicating which x ought to 
be chosen as a solution of (18). One way of doing this is by considering both the 
image vector x and the error vector e to be samples of random variables, denoted 
by X and EF, respectively. 

As an example of such an approach, let 4 denote a J-dimensional vector of 
real numbers and let V denote a J x J positive definite symmetric matrix of real 
numbers. We can define a function py over the set of all J-dimensional vectors of 
real numbers by 


1 
Px(x) = exp (-5 (x—p) VIO 1) (20) 


(27)//2 (det V)!/2 


This px is a probability density function of a random variable X on the set of all 
J-dimensional vectors of real numbers whose mean vector is j4, = 4 and whose 
covariance matrix is Vy = V.A random variable X defined in such a fashion is 
called a multivariate Gaussian random variable. 

Let us now consider the random variables X and F associated with x and e 
of (18) without assuming any special form for them. In any case, py is referred 
to as the prior probability density function, since px (x) indicates the likelihood of 
coming across an image vector similar to x. In CT it makes sense to adjust py to the 
area of the body we are imaging; the probabilities of the same picture representing 
a cross section of the head or of the thorax should be different. Based on py and 
PE, a reasonable approach to solving the discrete reconstruction problem is: given 
the data y, choose the image vector x for which the value of 


PE(y — Rx)px(x) (21) 


is as large as possible. Note that the second term in the product is large for vectors 
x that have large prior probabilities, while the first term is large for vectors x that 
are consistent with the data (at least if pz peaks at the zero vector). The relative 
importance of the two terms depends on the nature of py and pz. If py is flat (many 
image vectors are equally likely) and pz is highly peaked near the zero vector, 
then our criterion will produce an image vector x* that fits the measured data y in 
the sense that Rx* will be nearly the same as y. On the other hand, if pz is flat 
(large errors are nearly as likely as small ones) but py is highly peaked, our having 
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made our measurements will have only a small effect on our preconceived idea as 
to how the image vector should be chosen. The x* that maximizes (21) is called the 
Bayesian estimate. The discussion in this paragraph is quite general, since we have 
not assumed anything regarding the form of the random variables X and F. 

If we assume that both X and E are multivariate Gaussian, then maximizing (21) 
becomes relatively simple. In that case it is easy to see from (20) that, assuming that 
[4g is the zero vector, the x that maximizes (21) is the same x that minimizes 


(v= Ray ve (y — Rx) + (x — px)” Vy! (x — tx). (22) 


When more precise information regarding the mean vector jty is not available, 
one can use for it a uniform picture, with an estimated (based on the projection 
data) average value assigned to every pixel; how well this works out in practice is 
illustrated below in the section on Numerical Methods and Case Examples. We also 
illustrate there an alternative choice that is appropriate for cardiac imaging in which 
[4x is a time-averaged reconstruction. The noise model expressed by the first term 
of (22) is only approximate, but it is a reasonable accurate approximation of the 
effect of photon statistics in CT (see Section 3.1 of [23]). 

As representative examples of the series expansion methods for image 
reconstruction we now discuss thealgebraic reconstruction techniques (ART). 
All ART methods of image reconstruction are iterative procedures: they 
produce a sequence of vectors x, x,... that is supposed to converge to 
x*. The process of producing x“*) from x) is referred to as an iterative 
step. 

In ART, xt is obtained from x“ by considering a single one of the J 
approximate equations, see (17). In fact, the equations are used in a cyclic order. 
We use i, to denote k(mod J) + 1l;ie.,i9 = 1, i) = 2,..., t7-) = J, iy = 
1, i74; = 2,..., and we use 7; to denote the J-dimensional column vector whose 
jth component is r;,;. In other words, r; is the transpose of the ith row of R. (In 
what follows we assume that, for 1 <i < J, ||r; \|? = (r;,r;) 4 0, where, as usual, 
||e|| denotes the norm and (e, e) denotes the inner product.) An important point here 
is that this specification is incomplete because it depends on how we index the lines 
for which the integrals are estimated. As stated above, we assume that estimates 
of [% f] (€, @) are known for J pairs: (£1, 01),..., (€7, 0;). However, we have not 
specified the geometrical locations of the lines that are parametrized by these pairs. 
Since the order in which we do things in ART depends on the indexing i for the set 
of lines for which data are collected, the specification of ART as a reconstruction 
algorithm is complete only if it includes the indexing method for the lines, which we 
refer to as the data access ordering. We return to this point later on in this chapter. 

A particularly simple variant of ART is the following. 


x is arbitrary, 


xO) = xO 4 Op, (23) 
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Fig. 13 Demonstration of the method of (23) and (24) (with A = 1, for all k) for the simple 
case when J = J = 2 (illustration based on [25], Copyright 1976, with permission from Elsevier) 


where 


) 
in ANVIL X 
ol) = 00 2 — Vax) . (24) 
IIrix I 


with each A“ a real number, referred to as a relaxation parameter. It is easy to 
check that, for k > 0, if A“ = 1, then 


J 
k+1 
w= tea”, (25) 
= 


i.e., the i, th approximate equality is exactly satisfied after the kth step. This behavior 
is illustrated in Fig. 13 for a two-dimensional case with two equalities. 

This method has an interesting, although by itself not particularly useful, 
mathematical property. Let 


L={x|Rx = y}. (26) 
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A sequence x, x, x@,... generated by (23) and (24) converges to a vector x* 
in L, provided that L is not empty and that, for some ¢€; and ¢€ and for all k, 


0< 4 <1" =s <7, (27) 
Furthermore, if x° is chosen to be the vector with zero components, then 
I|x*|] < [ll]. (28) 


for all x in L other than x*. A proof of this can be found in Section 11.2 of [23]. 

The reason why this result is not useful by itself is that the condition that L is not 
empty is unlikely to be satisfied in a real tomographic situation. However, as it is 
shown in Section 11.3 of [23], it can be used to derive an alternative ART algorithm 
that is useful in real applications, as we now explain. 

Let us make the simplifying assumptions in (22) that Vy and Vg are both 
multiples of identity matrices of appropriate sizes. In other words, we assume that 
components of a sample of X — jty are uncorrelated, and that each component 
is a sample from the same Gaussian random variable; and we also assume that 
components of a sample of £ are uncorrelated and that each component is a sample 
from the same zero mean Gaussian random variable. We use s* to denote the 
diagonal entries of Vy and n? to denote the diagonal entries of Vz and let t = s/n. 
According to (22), the Bayesian estimate is the vector x that minimizes 


? ly — Rx||? + |x — wx. (29) 


Note that a small value of ¢ indicates that prior knowledge of the expected value of 
the image vector is important relative to the measured data, while a large value of 
t indicates the opposite. The following variant of ART converges to this Bayesian 
estimate, provided only that the condition expressed in (27) holds: 


u) is the J-dimensional zero vector, 
wel = nae 


30 
uktd = u&) + ce, , ( ) 
x k+) = xh) + tcr,,, 
where 
tp x(e\) _ 
ahh yw! (Viz ae )) Ui. a 


1+ 2? (Iriel? 


Note that both in (23) and in (30) the updating of x“) is very simple: we just 
add to x“) a multiple of the vector r;,- In practice, this updating of x can 
be computationally very inexpensive. Consider, for example, the basis functions 
associated with a digitization into pixels (10). Then r;,; is just the length of 


Tomography 821 


Fig. 14 A digital difference analyzer (DDA) for lines (reproduced from [23]) 


intersection of the ith line with the jth pixel. This has two consequences. First, most 
of the components of the vector r;, are zero. At most 2n — | pixels can be intersected 
by a straight line in an n x n digitization of a picture. Thus, of the n? components of 
rj,, at most 2n— 1 (and typically only about 7) are nonzero. Second, the location and 
size of the nonzero components of r;, can be rapidly calculated from the geometrical 
location of the i; th line relative to the n x n grid using a digital difference analyzer 
(DDA) methodology demonstrated in Fig. 14 (for details, see Section 4.6 of [23]). 
Thus, the projection matrix R does not need to be stored in the computer. Only one 
row of the matrix is needed at a time, and all information about this row is easily 
calculable. For this reason such methods are also referred to as row-action methods. 

We investigate this point further, since it is basic to the understanding of the 
computational efficacy of ART. Suppose that we have obtained, using a DDA, the 
list j;,..., jy of indices such that r;,,; = 0 unless j is one of the j;,..., jy. Then 
evaluation of Fe ‘ x) or of ||ri, ||? requires only U multiplications, which in our 
application is much smaller than J. The updating of x can be achieved by a further 


822 G.T. Herman 


U multiplications. This is because only those x; need to be altered for which j = j, 


for some u, 1 < u < U, and the alteration requires adding to a a fixed multiple of 
rj,,;- This shows that a single step of either of the ART algorithms described above 
is very simple to implement in a computationally efficient way. 

The ART are, in fact, a subclass of the class of projection methods, which have 
been demonstrated to be very effective in practice for solving convex feasibility 
problems with linear inequality constraints [6]. A recent advance in this direction 
is the development of the superiorization methodology [30] whose underlying idea 
is the following. There are many efficient iterative algorithms that produce feasible 
solutions for given constraints. Often the algorithm is perturbation resilient in the 
sense that, even if certain kinds of changes are made at the end of each iterative 
step, the algorithm still produces a feasible solution. This property is exploited in 
superiorization by using the perturbations to steer the algorithm to a solution that is 
not only constraints-compatible, but is also desirable according to an optimization 
criterion. This approach is applicable to many iterative procedures and optimization 
criteria. 


4 Numerical Methods and Case Examples 


Having seen that there is a variety of reconstruction algorithms, it is natural to 
ask for guidance as to when it is better to apply one rather than the others. 
Unfortunately, any general answer is likely to be misleading since the relative 
efficacy of algorithms depends on many things: the underlying task at hand, the 
method of data collection, the hardware/software available for implementing the 
algorithms, etc. The practical appropriateness of an algorithm under some specific 
circumstances needs experimental evaluation. 

We are now going to illustrate this by comparing, from certain points of view, 
the various reconstruction algorithms mentioned in the previous section. Except 
where otherwise stated, the generation of images and their projection data, the 
reconstructions from such data, the evaluation of the results, and the graphical 
presentation of both the images and the evaluation results were done within the 
software package SNARKO9 [12,37]. 

We studied a cross section of a human head that was reconstructed by CT 
(see Fig. 15). Based on this cross section we described a skull enclosing the brain 
with ventricles, two tumors, and a hematoma (blood clot) using five ellipses, eight 
segments of circles, and two triangles. The tumors were placed so that they are 
vertically above the blood clot in the display. We used SNARKO9 to obtain the 
density in each of 243 x 243 pixels of size 0.0752 cm. The resulting array of numbers 
is represented in Fig. 16. The nature of this display deserves careful discussion. The 
displayed values are linear attenuation coefficients jzz(x, y) at energy €e = 60keV 
of the appropriate tissue types measured in cm~!. Thus the values range between 
0 (background, can be thought of as air) and 0.416 (bone of the skull). However, 
the interesting part of the picture is inside the skull. The values there range from 
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Fig. 15 Central part of an X-ray CT reconstruction of a cross section of the head of a patient. This 
served as the basis for our piecewise-homogeneous head phantom (reproduced from [23]) 


0.207 (cerebrospinal fluid) to 0.216 (metastatic breast tumor). The small differences 
between these tissues would not be noticeable if we used black to display zero, white 
to display 0.5, and corresponding grayness for values in between. To see clearly 
the features in the interior of the skull, we use zero (black) to represent the value 
0.204 (or anything less) and 255 (white) to represent the value 0.21675 (or anything 
more). This way the small change in density by 0.001 corresponds to a change of 20 
in display grayness, which is visible. We did this to produce Fig. 16 and the displays 
of all the reconstructions of the head phantoms used as illustrations in this chapter. 

In Fig. 17a we show an actual brain cross section. The left half of the image shows 
a malignant tumor that has a highly textured appearance. In order to simulate the 
occurrence of a similarly textured object in our phantom we produced the phantom 
shown in Fig. 17b. Because of the medical relevance of imaging brains with such 
tumors, for the rest of this chapter we use the head phantom with this tumor added 
to it. (Due to our display method, it seems that there is a large range of values in the 
tumor. However, this is an illusion: the range of values in the tumor is less than 7 % 
of the range of values in the picture that is displayed in Fig. 16.) 

One problem with the phantoms as defined so far is that a brain is far from 
being homogeneous: it has gray matter, white matter, blood vessels and capillaries 
carrying oxygenated blood to and deoxygenated blood from the brain, etc. This is 
even more so for bone, whose strength to a large extent is derived from its structural 
properties. There are methods that can obtain remarkably accurate reconstruction 
of piecewise homogeneous objects, but their performance may not be medically 
efficacious when applied to CT data from real objects with local inhomogeneities. 
So as not to fall into the trap of drawing too optimistic conclusions from experiments 
using piecewise homogeneous objects, we superimposed on our head phantom 
a random local variation that is obtained by picking, for each pixel, a sample 
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Fig. 16 A piecewise-homogeneous head phantom (reproduced from [23]) 


Fig. 17 (a) An actual brain cross section with a tumor (image is reproduced, with permission, 
from the Roswell Park Cancer Institute website). (b) The head phantom of Fig. 16 with a “large 
tumor” added to it (reproduced from [23]) 


from a Gaussian random variable X with mean zy = | and standard deviation 
oy = 0.0025 and then multiplying the previously estimated linear attenuation 
coefficient at that energy level with that sample. In Fig. 18 we show the result of 
this. 
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Fig. 18 (a) A head phantom with local inhomogeneities with the 131st of the 243 columns 
indicated by a vertical line. (b) The densities along this column in the phantom (reproduced from 


[23]) 


A reconstruction is a digitized picture. If it is a reconstruction from simulated 
projection data of a test phantom, we can judge its quality by comparing it with the 
digitization of the phantom. Naturally, both the picture region and the grid must be 
the same size for the reconstruction and the digitized phantom. We now discuss how 
to illustrate and measure the resemblance between a reconstruction and a phantom. 
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Visual evaluation is of course the most straightforward way. One may display 
both the phantom and the reconstruction and observe whether all features in 
which one is interested in the phantom are reproduced in the reconstruction and 
whether any spurious features have been introduced by the reconstruction process. A 
difficulty with such a qualitative evaluation is its subjectivity, people often disagree 
on which of two pictures resembles a third one more closely. 

A more quantitative way of evaluating pictures is the following. Select a column 
of pixels that goes through a number of interesting features. For example, in our 
digitized head phantom the 131st of the 243 columns goes through the ventricles, 
both tumors, and the hematoma. In Fig. 18a we indicate this column. A way to 
evaluate the quality of a reconstruction is to compare the graphs of the 243 pixel 
densities for this column in the phantom (shown in Fig. 1 8b) and the reconstruction. 

It also appears desirable to use a single value that provides a rough measure of 
the closeness of the reconstruction to the phantom. We now describe two different 
methods of doing this. In our definition of these two picture distance measures we 
use fy and r,,, to denote the densities of the vth pixel of the uth row of the digitized 
test phantom and the reconstruction, respectively, and f to denote the average of the 
densities in the digitized test phantom. We assume that both pictures aren x n. Let 


noon non 1/2 
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(|x| denotes the absolute value of x.) These are often-used measures in the literature. 

These measures emphasize different aspects of picture quality. The first one, d, is 
a normalized root mean squared distance measure. A large difference in a few places 
causes the value of d to be large. Note that the value of d is 1 if the reconstruction 
is a uniformly dense picture with the correct average density. The second one, r, is 
a normalized mean absolute distance measure. As opposed to d, it emphasizes the 
importance of a lot of small errors rather than of a few large errors. Note that the 
value of r is 1 if the reconstruction is a uniformly dense picture with zero density. 

However, a collection of a few numbers cannot possibly take care of all the ways 
in which two pictures may differ from each other. Rank ordering reconstructions 
based on a few measures of closeness to the phantom can be misleading. We 
recommend instead a statistical hypothesis testing based methodology that allows 
us to evaluate the relative efficacy of reconstruction methods for a given task. 

This evaluation methodology considers the following to be the relevant basic 
question: given a specific medical problem, what is the relative merit of two (or 
more) image reconstruction algorithms in presenting images that are helpful for 
solving the problem? (Compare this with the alternative essentially unanswerable 
question: which is the best reconstruction algorithm?) Ideally, the evaluation should 
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be based on the performance of human observers. However, that is costly and 
complex, since a number of observers have to be used, each has to read many 
images, conditions have to be carefully controlled, etc. Such reasons lead us to use 
numerical observers instead of humans. The evaluation methodology consists of 
four steps: 


(i) Generation of random samples from a statistically described ensemble of 
images (phantoms) representative of the medical problem and computer simu- 
lation of the data collection by the device under investigation. 

(ii) Reconstruction from the data so generated by each of the algorithms. 

(iii) Assignment of a figure of merit (FOM) to each reconstruction. The FOM should 
measure the usefulness of the reconstruction for solving the medical problem. 

(iv) Calculation of statistical significance (based on the FOMs of all the reconstruc- 
tions) by which the null hypothesis that the reconstructions are equally helpful 
for solving the problem at hand can be rejected. 


We now discuss details. For relevance to a particular medical task, the steps must be 
adjusted to that task. The task for which comparative evaluations of various pairs of 
reconstruction algorithms are reported below is that of detecting small low-contrast 
tumors in the brain based on reconstructions from CT data. 

The ensemble of images generated for this task is based on the head phantom 
with a large tumor and local inhomogeneities. Note that this by itself provides 
us a Statistical ensemble because the local inhomogeneities are introduced using 
a Gaussian random variable. However there is an additional (for the task more 
relevant) variability within the ensemble that is achieved as follows. We specify 
a large number of pairs of potential tumor sites, the locations of the sites in a pair 
are symmetrically placed in the left and right halves of the brain. In any sample 
from the ensemble, exactly one of each pair of the sites will actually have a tumor 
placed there, with equal probability for either site. The tumors are circular in shape 
of radius 0.1 cm and with linear attenuation as for the meningioma in the original 
phantom. In Fig. 19a we illustrate one sample from this ensemble. Once a sample 
has been picked, we generate projection data for it by simulating a CT scanner, with 
all its physical inaccuracies as compared to the idealized Radon transform. (Such 
inaccuracies include: the finite number of measurements, statistical noise due to the 
finite number of X-ray photons used during the measurements, the hardening of the 
polychromatic X-ray beam as it passes through the body, the width of the detector, 
and the scattering of X-ray photons.) Further variability is introduced at this stage, 
since the data are generated by simulating noise due to photon statistics. In Fig. 19b 
we show a reconstruction from one such projection data set. The tumors are hard 
to see in this reconstruction, but that is exactly the point: we are trying to evaluate 
which of two reconstruction algorithms provides images in which the tumors are 
easier to identify. If we make the task too easy (by having large and/or high-contrast 
tumors), then all reasonable reconstruction algorithms would perform perfectly from 
the point of view of the task. On the other hand, if the task is too difficult (very small 
and very low-contrast tumors), then correct detection would become essentially a 
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Fig. 19 (a) Arandom sample from the ensemble of phantoms for the task-oriented comparison of 
reconstruction algorithms. (b) A reconstruction from noisy projection data taken of the phantom 
illustrated in (a) (reproduced from [23]) 


matter of luck, rather than of algorithm performance. Our ensemble was chosen to 
be in-between these extremes. The FOM that we chose to use is specific to the type 
of ensemble of phantoms that we have just specified. 

Given a phantom and one of its reconstructions, as in Fig. 19, we define the 
image-wise region of interest FOM (IROI) as 


B B 
>? (ai (b) — a, (6)) > (a? (b) — af ()) 
IROI = ihe al 
2 re 1 = ti y : Pp 1 - Pp / ; 
Dmoz ra) [Pilar yD ar 
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(34) 


The specification of the terms in this formula is as follows. For any digitized picture 
and for any potential tumor site, let the average density in that picture for that site be 
the sum over all pixels whose center falls within the site of the pixel densities divided 
by the number of such pixels. Let us number the pairs of potential tumor sites from 
1 to B, and let (for 1 < b < B) a? (b) (respectively, a? (b)) denote the average 
density in the phantom for site of the bth pair that has (respectively, has not) the 
tumor in it. We specify similarly a/ (b) (respectively, a’ (b)), for the reconstruction. 
The first thing to note about the resulting formula (34) is that the numerator and 
the denominator in the big fraction are exactly the same except that the numerator 
refers to the reconstruction and the denominator refers to the phantom. Thus, if 
the reconstruction is perfect (in the sense of being identical to the phantom) then 
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IROI = 1. Analyzing the contents of the numerator and the denominator, we see 
that they are (except for constants that cancel out) the mean difference between 
the average values at the sites with tumors and the sites without tumors, divided 
by the standard deviation of the average values at the non-tumor sites. It has been 
found by experiments with human observers that this FOM correlates well with the 
performance of people [49]. 

In order to obtain statistically significant results we need to sample the ensemble 
of phantoms and generate projection data a number (say C) of times. (For the 
experiments reported below we used C = 30.) Suppose that we wish to compare 
the task-oriented performance of two reconstruction algorithms. For 1 < c < C, 
let IROI(c) and IROI’(c) denote the values of IROI, as defined by (34), for the 
reconstructions by the two algorithms from projection data of the cth phantom. The 
null hypothesis that the two reconstruction methods are equally good for the task at 
hand translates into the statistical statement that each value of IROI'(c) — IROFP (c) 
is a sample of a continuous random variable D whose mean is 0. We have no idea 
of the shape of the probability density function pp of this random variable, but by 
the central limit theorem (see, e.g., Section 1.2 of [23]), for a sufficiently large C, 


s =) (IROI'(c) — IROP(c)) (35) 


Cc 
“=| 


c 


can be assumed to be a sample from a Gaussian random variable S with mean 0. This 
fact allows us to say (for details see Section 5.2 of [23]) that, at least approximately, 
S is a Gaussian random variable whose mean is 0 and whose variance is 


Cc 
Vs = 5> (IROI'(c) — IROP(c))”. (36) 
c=1 


It is a consequence of the null hypothesis that s is a sample from a zero-mean 
random variable. However, even if that were true, we would not expect our particular 
sample s to be exactly 0. Suppose for now that s > 0. This makes us suspect that 
in fact the first algorithm is better than the second one (for our task) and so the null 
hypothesis may be false. The question is: how significant is the observed value s 
for rejecting the null hypothesis? To answer this question we consider the P-value, 
which is the probability of a sample of S being as large or larger than s. If the 
null hypothesis were correct, we would not expect to come across an s defined by 
(35) for which the P-value is very small. Thus, the smallness of the P-value is a 
measure of significance for rejecting the null hypothesis that the two reconstruction 
algorithms are equally good for our task in favor of the alternative hypothesis that 
the first one is better than the second one. This is for the case when s > 0. If s < 0, 
then the P-value is the probability of a sample of S being as small or smaller than s 
and the alternative hypothesis is that the second algorithm is better than the first one. 

Having specified various methodologies for reconstruction algorithm evaluation, 
we now apply them to specific algorithms. Whenever we report on the performance 


830 G.T. Herman 


of an algorithm for the reconstruction of a single two-dimensional phantom, 
the phantom is the one shown in Fig. 18. For experiments involving statistical 
hypothesis testing, we use the ensemble illustrated by Fig. 19. In either case, 
the data collection geometry is the one described in Fig.9 with the number of 
source positions M = 720. Consequently, the angle mA shown in Fig.9 is 0.5m 
degrees. The source positions are equally spaced around a circle of radius 78cm. 
The distance of the source from the detector strip is 110.735cm. There are 345 
detectors, and the distance between two detectors along the arc of the detector strip 
is 0.10668 cm. We refer to this geometry of data collection as the standard geometry. 

The reconstruction algorithm estimates a digitization of the phantom from the 
projection data. Figure 20 shows the 243 x 243 digitization of the head phantom, a 
reconstruction by FBP from perfect projections (line integrals) for the geometry just 
described, and the values of the digitized phantom and the reconstruction along the 
131st column. The picture distance measures for this reconstruction are d = 0.0531 
and r = 0.0185. Even though the data are perfect, the reconstruction is not. This is 
because a picture is not uniquely determined by its integrals along a finite number 
of lines. The best that a reconstruction algorithm can do is to estimate the picture. 

There are interesting observations that one can make regarding this reconstruc- 
tion. One is that, generally speaking, the brain appears smoother in it than in the 
phantom. This is because the FBP algorithm that we use was designed to perform 
efficaciously on real data and it does some smoothing to counteract the effect of 
noise. Consequently, small variations due to inhomogeneity are also smoothed. The 
most noticeable features in the reconstruction that are not present in the phantom 
are the streaks that seem to emanate from straight interfaces between the skull 
and the brain. (Similar features are observable in the real reconstruction shown 
in Fig. 15.) Their presence can be explained by considering Radon’s formula (1), 
which expresses the distribution of the linear attenuation coefficient in terms of its 
line integrals. Consider an ¢ and a @ such that m(£, @) is the integral along a line 
that is very near to a straight edge between the skull and the brain. Due to the fact 
that attenuation is much larger for bone than for brain, numerical estimation of the 
partial derivative m, (€,@) from the discretely sampled projection data is likely to 
be inaccurate, introducing errors into the calculated reconstruction. Phantoms that 
lack such anatomical features should not be used for algorithm evaluation, since 
the resulting reconstructions do not indicate the errors that will occur in a real 
application in which the object to be reconstructed is likely to have such straight 
interfaces. This is illustrated in Fig. 21. 

The reconstructions shown in Figs. 20 and 21 are from “perfect” data; i.e., from 
line integrals based on the geometrical description of the phantoms. When data 
are collected by an actual CT scanner there are many physical reasons why the 
data so obtained can only provide approximations to such line integrals. In testing 
reconstruction algorithms we should use realistic projection data, which is what 
was done for the remaining two-dimensional reconstructions in this chapter. The 
exact method of simulated data collection (using SNARKO9 [12,37]) is described 
in Section 5.8 of [23], here we just give an outline. The data were collected 
for the head phantom shown in Fig. 20a according to the standard geometry. For 
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Fig. 20 (a) Head phantom (the same as Fig. 18a). (b) Its reconstruction from “perfect” data 
collected for the standard geometry. (c) Line plots of the 131st column of the phantom (light) 
and the reconstruction (dark) (reproduced from [23]) 


photon statistics we chose an average of million X-ray photons originating in the 
direction of each detector during the scanning of the head. A realistic spectrum of 
the polychromatic X-ray source was also simulated. The focal spot of the X-ray 
source was assumed to be a point, but the detectors were assumed to have width of 
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Fig. 21 (a) A simple head phantom without straight edges between bone and brain. (b) Its 
reconstruction from “perfect” data collected for the standard geometry. In this reconstruction there 
are no false features of the kind that emanate from the straight edges in Fig. 20b (reproduced from 
[23]) 


0.10668 cm (i.e., there are no gaps between the detectors). It was assumed that the 
number of scattered photons that are counted during the measurements is 5 % of the 
number of unscattered photons that are counted. The data so obtained was corrected 
for beam-hardening, to provide us with an estimate of the monochromatic projection 
data. The outcome of this correction is what we refer to as the standard projection 
data. For the experiments involving statistical evaluation, the same assumptions 
were made except that the phantom was randomly selected from the previously 
described ensemble; for an example, see Fig. 19a. Our illustrations are restricted 
to demonstrating the effects of various choices that can be made in ART and the 
comparison of ART with FBP. 

We start with the variant of ART described by (23) and (24). We choose x to 
represent a uniform picture, with the estimated (based on the standard projection 
data) average value of the phantom assigned to every pixel. (The estimation of the 
average value from projection data is described in Section 6.4 of [23].) 

We first show that the order of equations in the system (the data access 
ordering discussed in the previous section) can have a significant effect on the 
practical performance of the algorithm, especially on the early iterates. With 
data collection such as the geometry depicted in Fig. 9, it is tempting to use the 
sequential ordering: access the data in the order g(—NA,0), g((—N + 1)A,0),..., 
g(NA,0), g(—-NA, A), g((-N+1)A,A),...,9(NA,A),...,..., 9(- NA, (WM 
1)A), g(-—N + 1)A, (M — 1)A),..., g(NA,(M — 1)A), where g(o, 8) denotes 
here the measured value of what is mathematically defined in (5). However, this 
sequential ordering is inferior to what is referred to as the efficient ordering in which 
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Fig. 22 Values of the picture distance measure r for ART reconstructions from the standard 
projection data with sequential ordering (light) and efficient ordering (dark), plotted at multiples 
of J iterations (reproduced from [23]) 


the order of projection directions mA and, for each view, the order of lines within 
the view is chosen so as to minimize the number of commonly intersected pixels 
by a line and the lines selected recently. This can be made mathematically precise 
by considering the decomposition into a product of prime numbers of M and of 
2N + 1 [27]. SNARKO9 [12,37] calculates the efficient order, but this is only useful 
if both M and of 2N + 1 decompose into several prime numbers, as is the case 
for our standard geometry for which M = 720 =2x2x2x2x3x3x5 and 
2N +1 = 345 =3 x5 x 23. While the sequential ordering produces the sequences 
m = 0,1,2,3,4,... andn = 0,1,2,3,4,..., the efficient ordering produces the 
sequences m = 0,360, 180,540, 90,... and = 0,115,230, 23, 138,... These 
changes in data access ordering (keeping all other choices the same) translate 
into faster initial convergence of ART, as is illustrated in Fig.22 by plotting the 
picture distance measure r of (33) against the number of times the algorithm cycled 
through all the data (all J equations). To produce this illustration we used blob basis 
functions and A“) = 0.05, for all k. While it is clearly demonstrated that initially 
r gets reduced much faster with the efficient ordering, for the standard projection 
data it does not seem to matter much, since both orderings need about five cycles 
through the data to obtain a near-minimal value of r. In other applications in which 
the number of projection directions is much larger (e.g., in the order of 10,000 
as is often the case in electron microscopy), one cycle through the data using the 
efficient ordering yields about as good a reconstruction as one is likely to get, but the 
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Fig. 23 Reconstructions from the standard projection data using ART. (a) ART with blobs, 4 = 
0.05, 5/th iteration and efficient ordering. (b) ART with blobs, A® = 0.05, 5/th iteration and 
sequential ordering. (c) ART with pixels, A“) = 0.05, 5/th iteration and efficient ordering. (d) 
ART with blobs, A = 1.0, 2/ th iteration and efficient ordering (based on Fig. 11.4 of [23]) 


sequential ordering needs several cycles through the data. In addition, the efficacy 
of the reconstruction produced by the efficient ordering may very well be superior 
to that produced by the sequential ordering. 

This is illustrated in Fig.23 and Table |. The reconstructions produced by the 
efficient and sequential orderings after five cycles through the data (the images 
of x©”) are shown in Figs. 23a and b, respectively. Visually there is hardly any 
difference between them. This is confirmed by the picture distance measures in 
Table 1, they are only slightly better for the efficient ordering than for the sequential 
ordering. On the other hand, the execution time (within the SNARKO9 [12, 37] 
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Table 1 Picture distance measures and timings (in seconds, of the implementations in SNARKO9) 


for the reconstructions in Figs. 23 and 25 


Reconstruction in t IROI 

Figure 23a 163.7 0.1794 
Figure 23b 148.9 0.1624 
Figure 23c 29.2 0.1592 
Figure 23d 66.2 0.1076 
Figure 25b 8.7 0.1677 


The last column reports the values, produced by a task-oriented evaluation experiment, of the IROI 
for the various algorithms (based on Table 11.1 of [23]) 
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Fig. 24 Values of the picture distance measure r for ART reconstructions from the standard 


projection data with pixels (Jight) and blobs (dark), plotted at multiples of J iterations (reproduced 
from [23]) 


environment) is somewhat less for the sequential ordering. However, the task- 
oriented evaluation is unambiguous in its result: the [ROI is larger for the efficient 
ordering and the associated P-value is less than 10°. This means that we can reject 
the null hypothesis that the two data access orderings are equally good in favor of the 
alternative hypothesis that the efficient ordering is better with extreme confidence. 
Next we emphasize the importance of the basis functions. In Fig. 24 we plot the 
picture distance measure r against the number of times ART cycled through all the 
data, where we kept all other choices the same (in particular, efficient data access 
ordering and A“ = 0.05, for all k). The two cases that we compare are when the 
basis functions are based on pixels (10) and when they are based on blobs (12). The 
results are impressive: as measured by r, blob basis functions are much better. The 
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result of the 5/th iteration of the blob reconstruction is shown in Fig. 23a, while 
that of the 5/th iteration of the pixel reconstruction is shown in Fig. 23c. The blob 
reconstruction appears to be clearly superior. In Table 1 we see a great improvement 
in the picture distance measure r but not in d. This reflects the fact, not visible 
in our display mode, that there are a few but relatively large errors in the blob 
reconstruction near the edges of the bone of the skull. From the points of view of the 
task-oriented figure of merit IROI, ART with blobs is found superior to ART with 
pixels with the relevant P-value less than 10~!°. As implemented in SNARKO9, 
ART with blobs requires significantly more time than ART with pixels, but there 
exist more sophisticated implementations of ART with blobs that are much faster. 

Underrelaxation is also a must when ART is applied to real, and hence imperfect, 
data. In the experiments reported so far A“) was set equal to 0.05 for all k. If we 
do not use underrelaxation (that is, we set A” to 1 for all k), we get from the 
standard projection data the unacceptable reconstruction shown in Fig. 23d. Note 
that in this case we used the 2/ th iterate, further iterations give worse results. The 
reason for this is in the nature of ART: after one iterative step with A”) = 1, the 
associated measurement is satisfied exactly as shown in (25) and so the process 
jumps around satisfying the noise in the measurements. Underrelaxation reduces the 
influence of the noise. The correct value of the relaxation parameter is application 
dependent; the noisier the data the more we should be underrelaxing. Note in Table | 
that the figure of merit IROI produced by the task-oriented study for the case without 
underrelaxation is much smaller than for the other cases. 

Now we compare the best of our ART reconstruction (Fig. 23a, reproduced in 
Fig. 25a) with one produced by a carefully selected variant of FBP, see (6)—(9). For 
comparison, we show in Fig. 25b the reconstruction from our standard projection 
data obtained by FBP for divergent beams with linear interpolation and sinc window 
(also called the Shepp—Logan window, see [56]). For details of the meanings of 
these choices and the reasons for them, see Chapter 10 of [23]. The visual quality is 
similar to the best ART reconstruction. According to the picture distance measures 
in Table 1, ART is superior to FBP, and the same is true according to IROI with 
extreme significance (the P-value is less than 10~!%). This experiment confirms 
the reports in the literature that ART with blobs, underrelaxation and efficient 
ordering generally outperforms FBP in numerical evaluations of the quality of the 
reconstructions. 

One thing though is indisputable: the ART with blob reconstruction took nearly 
19 times longer than FBP. However, this should not be the determining factor, 
especially since the implementation of ART with blobs in SNARKO9 is far from 
optimal and can be greatly improved. An advantage of ART over FBP is its 
flexibility. Even though until now we have reported its application only to data 
collected according to the standard geometry, ART is capable of reconstructing from 
data collected over any set of lines, as we soon demonstrate by an example of using 
ART for helical CT. FBP-type algorithms need to be reinvented for each new mode 
of data collection. 
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Fig. 25 Comparison of reconstructions from the standard projection data using (a) ART (the same 
as Fig. 23a) and (b) FBP (based on Fig. 11.4 of [23]) 


We now switch over to demonstrating the ART algorithm specified in (30) 
and (31). As stated before, that algorithm converges to the Bayesian estimate that is 
the minimizer of (29), provided that the condition expressed in (27) holds. This is 
the case if we set A“) = 0.05, for all k, which is what we chose for the experiments 
on which we now report. The other choices that we made are blob basis functions, 
efficient ordering and that, in (29), t = 10 and jzy represents a uniform picture with 
the estimated average value of the phantom assigned to every component. 

There are alternative methods in the literature for minimizing (29), a particularly 
popular one is the method of conjugate gradients (CG); for a description of it that 
is appropriate for our context, see Section 12.5 of [23]. The CG method is also an 
iterative one, but one in which all the data are considered simultaneously in each 
iterative step. For this reason, the time of one iterative step of the CG method is 
approximately the same as that needed by ART for one cycle through all the data. 
In Fig. 26 we show a comparison of the picture distance measure r for CG and for 
ART. 

Figure 26 and the picture distance measures in Table 2 imply that the quality 
of the reconstruction obtained by the 20th iterate of the conjugate gradient method 
should be as good as that obtained by the 5/th iterate of additive ART. However 
this is not really so, as can be seen by looking at the reconstructed image in 
Fig. 27b. Indeed it needs another 20 iterations of the conjugate gradient method 
before the visual quality of the reconstruction matches that of the ART of (30) 
and (31) after 5/ iterations, shown in Fig. 27a. So (for the standard projection data) 
the conjugate gradient method is not as fast as ART. This slower convergence of 
conjugate gradients relative to ART seems to be shared by other series expansion 
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Fig. 26 Values of the picture distance measure r for reconstructions from the standard projection 
data using the conjugate gradient method (light) and ART (dark), plotted for comparable 
computational costs (reproduced from [23]) 


Table 2 Picture distance measures and timings (in seconds) for the reconstructions that mini- 
mize (29) 


Algorithm d r t 
ART, 5/ th iterate 0.0878 0.0374 | «(166.5 
Conjugate gradient method, 20th iterate 0.0799 | (0.0387 489.1 


Based on Table 12.2 of [23] 


reconstruction methods that use all the data simultaneously in each iteration; see, 
for example, [58]. 

If we wish to reconstruct a three-dimensional body by the methods discussed till 
now, the only option available to us is to reconstruct the body cross section by cross 
section and then stack the cross sections to form the three-dimensional distribution. 
This may cause a number of problems, the most important of which are associated 
with time requirements. During the time needed to collect all the data, the patient 
may move, causing a misalignment between the cross sections. More basically, in 
moving organs such as the heart, changes in the organ over time are unavoidable, 
and it is usually not possible to collect data for all cross sections simultaneously. 

Sometimes, it is actually the change in the object over time that is the desired 
information. If we wish to see cardiac wall motion, then it is essential that we 
reconstruct the whole three-dimensional object at short time intervals. One may 
consider this as a four-dimensional (spatio-temporal) reconstruction. One approach 
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Fig. 27 Reconstructions from the standard projection data using iterative methods that minimize 
(29). (a) ART, 5/ th iterate. (b) Conjugate gradient method, 20th iterate (based on Fig. 12.2 of [23]) 


to obtaining reconstructions of dynamically moving objects, such as the heart, from 
data that can be collected by helical CT (see Fig. 8) is to assume that the movement 
is cyclic. Assuming also that there exists a way of recording where we are in the 
cyclic movement as we take the 2D views of the moving 3D object, it is possible 
to bin the views into subsets such that all views that are binned into any one of the 
subsets have been taken at approximately the same phase of the cyclic movement, 
and so they are views of approximately the same (time frozen) 3D object. In the 
case of the heart this can be done by recording the electrocardiogram and noting on 
it the times when views have been taken. These views can then be binned, after the 
fact, according to the phases of the cardiac cycle. 

We complete this section by giving a summary of such experiments, details can 
be found in Chapter 13 of [23]. The reconstructions were done by ART (here we 
made good use of the fact that ART does not require any particular arrangement 
of the lines for which the data were collected), using three-dimensional blobs [41] 
as the basis functions. We designed a phantom of the human thorax based on the 
description of the so-called FORBILD thorax phantom. We added to that stationary 
phantom two dynamically changing spheres representing the myocardium and a 
single contrast material filled cavity. We assumed that we are interested in this 
phantom at 24 equally spaced (in time) phases of the cardiac cycle. The first row of 
Fig. 28 shows a central cross section of this dynamic phantom at the two extremes 
of the 24 phases. 

Projection taking was done by integrating the density of the phantom along 
lines between the X-ray source position and detectors in a two-dimensional array. 
For every source position, data were collected for 384 equally spaced detectors 
in each of 16 rows in the array. The size of each detector was assumed to be 
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Fig. 28 The central cross section of the thorax phantom at the two extreme phases of the cardiac 
cycle. First row: the phantom. Second row: reconstruction from data collected at the time when 
the heart was in the appropriate phase after five cycles of simple ART. Third row: reconstruction 
from data collected at the time when the heart was in the appropriate phase after three cycles of 
Bayesian ART initialized with the reconstruction by two cycles of simple ART (based on Fig. 13.1 


of [23]) 
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0.425 x 0.425cm. Data were collected (i.e., the pulsing of the X-ray source was 
simulated) at every 0.0015 second, using a total of 8,400 pulses. The number of turns 
of the helix in which the X-ray source moved during the data collection was 30. The 
radius of the helix was 57cm, and the total movement parallel to the axis of the 
helix was 17.28 cm. The distance from the source to the detector array was 104cm. 
Integrals of the density were collected for J = 51,609,600 rays (8,400 pulses 
times 16 rows of 384 detectors). Detector area and the effect of photon statistics 
were also simulated. The numbers used in this paragraph are not inappropriate 
for helical CT, but a state-of-the-art helical CT scanner would have more and 
smaller detectors and would be pulsed more frequently. In all our experiments we 
used J = 2,153,935 three-dimensional blobs to describe the reconstructed three- 
dimensional distributions. 

In the first experiment we reconstructed the 24 phases of the cardiac cycle 
independently of each other. This was done by subdividing all the projection data 
into 24 subsets, each corresponding to one of the phases. A ray sum was put into a 
particular subset if it was collected due to a pulsing of the X-ray source at a time 
nearer to the central time for that phase than to the central time of any other phase. 
This results in a number of consecutive pulses producing data for the same phase and 
then there is a relatively large gap before the collected data are again used for that 
phase. This very nonuniform mode of data collection results in unacceptably bad 
reconstructions, two of which are demonstrated in the second row of Fig. 28. These 
reconstructions were produced using the simple ART of (23) and (24) with the three- 
dimensional blob basis functions, with all components of x° given the estimated 
average value based on the projection data, all A“) = 0.05 and an efficient ordering. 
The results are shown at the end of the fifth cycle through the data associated with 
the particular phase of the cardiac cycle. 

In the second experiment we used the other extreme: all the data were combined 
into a single projection data set, without any attention paid to the phases of 
the cardiac cycle. Because of the stationarity of most of the phantom and the 
overabundance of the projection data, we get (using the same choices for ART 
as in the previous paragraph) reconstructions that are good overall, but naturally 
the movement of the heart is blurred out due to the various views used in the 
reconstruction having been taken all through the cardiac cycle. We note that in this 
case there is no need to cycle through the data five times: the reconstruction at the 
end of the second cycle through the data is just about indistinguishable from the 
reconstruction at the end of the fifth cycle through the data. 

However, our aim here is to see the dynamic changes in the heart. This can be 
achieved by using the Bayesian approach of (30) and (31). We selected in (29) jury 
as the reconstruction obtained at the end of the second ART cycle through all the 
data as described in the previous paragraph and ¢ = 0.8. For each separate phase of 
the cardiac cycle, we used the algorithm specified by (30) and (31) for a further three 
cycles through the data that are associated with that particular phase. The relaxation 
parameter was again the constant 0.05. The results, for the two extreme phases of the 
cardiac cycle, are shown in the last row of Fig. 28. Here the overall reconstruction 
of the thorax is quite good and, at the same time, one can observe that the heart is 
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dynamically changing. With a state-of-the-art helical CT scanner (that would have 
more and smaller detectors and would be pulsed more often) we would get even 
better reconstructions. 


5 Conclusion 


Tomography is the process of producing an image of a distribution from estimates 
of its line integrals along a finite number of lines of known locations. There are a 
number of mathematical approaches to achieve this and we discussed and illustrated 
some of them. Of the investigated approaches, we found the performance of the 
method referred to as ART with blobs particularly good, especially if it is used with 
the appropriate data access ordering and relaxation parameters. 

We subdivided the recommended readings into categories. For additional relevant 
information see [23] that has 280 references, 83 of which have been published since 
2005. 

Books related tomography [2, 7, 17-19, 23, 24, 29, 33, 50,51, 59]. 

Papers on transform reconstruction methods and their applications [3, 9, 10, 13- 
15,28, 34, 35,52, 53,56]. 

Papers on series expansion reconstruction methods and their applications [4, 5, 
16, 20, 25, 27, 31,32, 39-41, 44-46, 54, 57]. 

Papers on comparison of reconstruction methods [6, 12, 21, 30, 36, 37, 43, 47— 
49,55, 58]. 

Papers on three-dimensional display of reconstructions [1, 8, 26, 42]. 


Cross-References 
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Large-Scale Inverse Problems in Imaging 

Linear Inverse Problems 

Mathematical Methods in PET and SPECT Imaging 
Mathematics of Electron Tomography 

Regularization Methods for Ill-Posed Problems 

Mathematics of Photoacoustic and Thermoacoustic Tomography 
Variational Methods in Shape Analysis 
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Abstract 


Several limited data problems in tomography will be presented in this chapter, 
including ones for X-ray tomography, electron microscopy, and radar imaging. 
First, reconstructions from limited data will be evaluated to observe their 
strengths and weaknesses. Then, the basic analytic properties of the transforms 
will be presented. The concept of microlocal analysis will be introduced to 
make the notion of singularity precise. Finally, the microlocal properties of the 
tomographic transforms are given and then used to explain the observed strengths 
and limitations of the reconstructions. This will show that these limitations are 
intrinsic to these limited data problems themselves. 


1 Introduction 


In this chapter, a range of tomography problems are introduced, including X- 
ray imaging, limited data problems, electron microscopy, and radar imaging. The 
goal is to recover the singular features of the medium or object rather than to 
develop exact inversion formulas. Toward this end, microlocal analysis is used to 
understand the strengths and limitations inherent in these tomography problems. 
Microlocal analysis aids researchers in understanding those singular features that 
can be stably recovered, which could be very important when only limited or partial 
data are available. Furthermore, it helps explain the presence of artifacts in certain 
image reconstruction methods. In some cases, it might help to distinguish the true 
singularities from the false ones. These are the themes emphasized in this chapter. 

In Sect.2, the tomography problems are introduced. Reconstructions are pre- 
sented for each problem and reconstruction quality is evaluated with the goal of 
finding strengths and limitations for each method. In Sect. 3, some basic properties 
of tomographic transforms are introduced. Microlocal analysis is introduced in 
Sect. 4. Finally, several applications in tomography and radar imaging are given in 
Sect. 5 with the goal of emphasizing the microlocal properties of these transforms. 
This powerful tool is used to clarify the strengths and limitations that are really 
intrinsic to the data. 


2 Motivation 


Several modalities in tomography, including X-ray tomography, limited data tomog- 
raphy, electron microscope tomography, and synthetic-aperture radar (SAR), are 
introduced in this section. For each problem, some history is given and then 
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reconstructions using such data are evaluated for strengths and weaknesses. The 
goal of this section is to observe, for each problem, object features that are well 
reconstructed and features that are not. These reconstructions are provided to 
motivate the study of microlocal analysis, which will be used in Sect. 5 to explain 
these reconstructions. 


X-Ray Tomography (CT) and Limited Data Problems 


In the 1970s, X-ray tomography revolutionized diagnostic medicine. For the first 
time, doctors were able to get clear and accurate pictures of the inside of the body 
without doing exploratory surgery. One part of this story began in the early 1960s. 
At that time, Allan Cormack consulted as a medical physicist at the Groote Schuur 
hospital in Cape Town, South Africa, and he checked whether X-ray machines were 
calibrated properly. He felt that there should be more information in the X-ray data 
than just what is obtained from single pictures, which project all organs onto the 
same plane, and he believed that X-rays could be used more effectively. He posited 
that if one takes X-ray images from multiple directions, one should be able to piece 
together the internal structure of the body. He then developed two algorithms [10, 
11] for the problem. To give a proof of concept, he built a prototype scanner that 
showed his second algorithm was effective. Along with Godfrey Hounsfield of EMI 
in England, he received the 1979 Nobel Prize in Medicine. You can read more about 
him in the excellent biography [97]. 

X-ray CT is now used routinely in medicine and in industrial nondestructive 
testing, and it allows doctors to image the internal structure of the body without 
exploratory surgery. The basic physics and mathematical model are now described. 
Let £ be a line along which X-rays travel, and for x € £, let J(x) be the intensity 
(number of photons) at the point x. Let f(x) be the attenuation coefficient of the 
body at x. For monochromatic light, f is proportional to the density at x, and by 
using a scale factor, they become the same. Beer’s law [66] states that the decrease in 
intensity at x is proportional to the intensity /(x), and the proportionality constant 
is — f(x): 

dl 


a —f (x)I(x). (1) 


This makes sense heuristically because the more dense the material at x (i.e., the 
larger f(x) is), the more the beam is attenuated and the greater the decrease of J 
at x. Equation (1) is a simple differential equation for 7 that can be solved using 
separation of variables. If Jp is the intensity at the X-ray emitter — the point x9 € 
—and J, is the intensity at the detector, x, € £, then one can integrate (1) to find 


I ml 
n(?) =/ fisyax =f flsyar. 


This leads to the definition 


Rif (0) = i, _ f(s) ax 
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where in this case, dx is the arc length measure on £. The transform ,, was studied 
by the Austrian mathematician Johann Radon [84] in the early twentieth century 
because it was intriguing pure mathematics. This transform is called the Radon line 
transform (or X-ray transform). 

To proceed mathematically, more notation is given. Let w € S! and let p € R. 
Then, the line 


L(w, p) = {x €R? : x-w= p} (2) 


is perpendicular to w and contains pw. Sometimes it will be useful to let w be a 
function of polar angle g € R, 


w(~) = (cos(g), sin(g)) . 


In this parameterization 


Ri flo.p)= | 


f(x) dx = / f (po + tat) dt (3) 
xEl(w,p) teR 
where w+ is the unit vector 17/2 radians counterclockwise from w. This integral is 
defined for f € C, (R*), and in fact R, is continuous in a number of norms (see 
section “Continuity Results for the X-Ray Transform’). The basic properties of this 
transform are proven in Sect. 3. 

First, consider the forward problem and a simple case that will show in a naive 
sense how the X-ray transform detects object boundaries. 


Example 1. Let f be the characteristic function of the unit disk in R?. Then, using 
the Pythagorean Theorem, one sees that 


2V1—p? |p|<1 


Rr f(@, p) = : 
LI (@, p 0 ioe 


(4) 


The function Rz f(@, p) in (4) is smooth except at p = +1, that is, except for 
lines £(@, +1) as can be seen from Fig. 1. The data are not smooth at those lines and 
those lines are tangent to the boundary of the disk. This suggests that lines tangent 
to boundaries give special information about the specimen. In Sect. 4, the reader will 
discover what is mathematically special about those lines, and this will be related 
back to limited data tomography in Sect. 5. 


For complete data, that is, data over all lines through the object, good reconstruc- 
tion methods such as filtered backprojection (Theorem 9) are effective to reconstruct 
from X-ray CT data. 

However, one cannot obtain complete data in many important tomography 
problems. These are called limited data tomography problems, and several important 
ones are now described. The goal at this point is to observe how the reconstructions 
look compared to the original objects. 
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Fig. 1 This graph shows the y 
calculation of the Radon 


transform in (4). The unit 

disk is above the graph. For f=1 

|p| < 1, the Pythagorean 

theorem shows that the length + 
of the intersection of £(@, p) P 
and the disk is 2\/1 — p? f=0 


Here are some guidelines as you read this section. For each problem and 
reconstruction, conjecture what is special about the object boundaries that are well 
reconstructed (and those that are badly reconstructed) in relation to the limited data 
set used. 


Exterior X-Ray CT Data 

Exterior CT data are data for lines that are outside an excluded region. Typically, 
that region is a circle of radius r > 0, so lines €(w, p) for |p| > r are in the data 
set. Theorem 5 in the next section shows that compactly supported functions can be 
uniquely reconstructed outside the excluded region from exterior data. 

The exterior problem came about in the early days of tomography for CT scans 
around the beating heart. In those days, a single scan of a planar cross section could 
take several minutes, and movement of the heart would create artifacts in the scan. 
If an excluded region were chosen to contain the heart and be large enough so the 
outside of that region would not move, then data exterior to that region would be 
usable. However, scanners soon began to use fan beam data (see section “Fan Beam 
and Cone Beam CT”), and data could be acquired much more quickly. If the data 
acquisition is timed (gated), then data are acquired while the heart is in the same 
position over several heartbeats. Because more data can be taken more quickly with 
fan beam data, the heart can now be imaged using newer scanners, and movement 
of the heart is not as large a problem. 

Exterior data are still important for imaging large objects such as rocket shells. 
Even with an industrial CT scanner, the X-rays will not penetrate the thick center 
of the rocket [88]. However, they can penetrate the outer rocket shell, and this gives 
exterior data. 

One can recover functions of compact support from exterior data, at least outside 
the excluded region (see Theorem 5). Effective inversion methods were developed 
for exterior data by researchers including Bates and Lewitt [3], Natterer [65], 
Quinto [77,79], and a stability analysis using singular value decompositions was 
done in [58]. 
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Fig. 2 Exterior reconstruction. Phantom (/eft) and reconstruction (right) from simulated data [77, 
©IOP Publishing. Reproduced by permission of IOP Publishing. All rights reserved]. The outer 
diameter of the annulus is 1.5 times the inner diameter 


Figure 2 is a reconstruction from exterior data: integrals are given over lines 
that do not meet the black central disk. The reconstruction method uses a singular 
value decomposition for the exterior Radon transform that includes a null space; it 
recovers the component of the object in the orthogonal complement of the null space 
and does an extrapolation to recover the null space component [77]. 

Note how some boundaries of the small circles are clearly reconstructed and 
others are not. In this case, how can you describe the boundaries that are well 
reconstructed in relation to the data set? Another question is whether the fuzzy 
boundaries are fuzzy because the algorithm is bad or could there be an additional 
explanation. 

Allan Cormack’s first algorithm [10] solved the exterior problem, but the 
algorithm was not numerically effective. The integrals in his algorithm were difficult 
to evaluate numerically with any accuracy because the integrand grew too rapidly. 
Other mathematicians tried to improve this method, but it was difficult. Because of 
this problem, Cormack developed a second method that uses full data and that gave 
good reconstructions [11]. 

It would be useful to know if limitations of Quinto’s and Cormack’s algorithms 
are problems with their algorithms or reflect something intrinsic to this limited data 
problem. 


Limited Angle Data 

Limited angle tomography is a classical problem from the early days of tomography 
(3,60, 61]. In this case, data are given over all lines in a limited range of directions, 
or data for {(w(g), p) : g € (—®,®@), p € R} where ® € (0,7/2). One can 
uniquely recover compactly supported functions from limited angle data, but this is 
not true for arbitrary functions (see Theorem 3). 

Limited angle tomography is used in certain luggage scanners in which the X- 
ray source is on one side of the luggage and the detectors are on the other, and 
they move in opposite directions. Limited angle data are used in important current 
problems including dental X-ray scanning [68] and tomosynthesis (a tomographic 
technique to image breasts using transmitter and receiver that move on opposite 
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sides of the breast) [72]. Other algorithms were developed for this problem such as 
[12, 26, 49, 68]. 

The reconstruction in Fig. 3 is from limited angle data. Data are taken over all 
lines £(w(g), p) for p € Rand g ¢€ [—7/4, 1/4]. 

The algorithm used in this reconstruction is a truncated filtered backprojection 
(FBP) algorithm which is given in (26). Some boundaries in this reconstruction are 
well reconstructed and others are not. How do these boundaries relate to lines in the 
data set? How do the streaks in the reconstruction relate to the data set? 


Region of Interest (ROI) Data 

In region of interest tomography, one chooses a subset of the object, called a region 
of interest (ROI), to reconstruct. ROI data consist of all lines that meet this region, 
and the ROI problem is to reconstruct the structure of the ROI from these data. Such 
data are also called interior data (and the interior problem). ROI CT is important in 
the CT of small parts of objects, so-called micro-CT [18, p. 460]. Other algorithms 
in ROI CT include [104] (when one knows the value of a function in part of the 
interior) and [105] (when the density is piecewise constant in the ROJ), as well as 
algorithms including [52]. A singular value decomposition was developed for this 
problem in [63]. 

ROI CT is useful for medical CT and industrial nondestructive evaluation in 
which one is interested only in a small region of interest in an object, not the entire 
object. An advantage for medical applications is that ROI data gives less radiation 
than with complete data. 

Lambda tomography [17], [18] is one important algorithm for ROI CT which will 
be described in section “Filtered Backprojection (FBP) for the X-Ray Transform,” 
and the ROI reconstruction presented here uses this algorithm. The data are severely 
limited — they include only lines near the disk and the ROI transform is not injective 
(see Theorem 6), so why do the reconstructions look so good? 


Limited Angle Region of Interest Tomography 

In this modality data are given over lines in a limited angular range and that are 
restricted to pass through a given ROI. It comes up in single axis tilt electron 
microscopy (ET) (see Oktem’s chapter in this book [71]). However, in general, ET 
is better understood as a three-dimensional problem, and this will be presented in 
the next paragraph. 


Electron Microscope Tomography (ET) Over Arbitrary Curves 


Now a full three-dimensional problem, electron microscope tomography (ET), is 
considered. The notation is as in Oktem’s chapter in this book [71], which has 
detailed information about the physics, biology, model, and mathematics of ET. 
A reconstruction of a simple 3D phantom — the union of the following disks: with 
center (0, 0, 0), radius 1/2; center (0, 0, 1), radius 1/2; center (1, —1, 1), radius 1/4; 
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Fig. 4 Reconstruction from conical tilt data. Cross section with the x — y plane of the phantom 
described in this section (left), £, reconstruction (center, see Eq. (28)) and Ls reconstruction 
(right, see Eq. (29)). The center of the cross section is the origin, and the range in x and y is from 
—2 to 2. This research was done with Sohhyun Chung and Tania Bakhos while they were Tufts 
undergraduates [82, ©Scuola Normale Superiore, Pisa. Reproduced by permission with Scuola 
Normale Superiore, Pisa, all rights reserved] 
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Fig. 5 Reconstruction from conical tilt data. Cross section of phantom in the plane x = —y 
(left) and £, reconstruction in that plane (right). The x — y plane cuts the picture in half with 
a horizontal line [82, ©Scuola Normale Superiore, Pisa. Reproduced by permission with Scuola 
Normale Superiore, Pisa, all rights reserved] 


and center (—1, 1, 1/2), radius 1/4 —is analyzed. The disks above the x — y plane 
have density two and the others have density one. 

Conical tilt ET data is described in section “Algorithms in Conical Tilt ET” and 
Oktem’s chapter in this book [71]. In this example, line integrals are given over all 
lines in space with angle a = 2/4 with the z—axis. Reconstructions are given from 
two algorithms that are described in section “Electron Microscope Tomography 
(ET) Over Arbitrary Curves.” The operators are £, (given in Eq.(28)) and Ls 
(given in Eq. (29)). 

Artifacts are added in the £, reconstruction in Fig. 4 and in Fig. 5 which shows 
the plane containing the centers of the disks and the z—axis (axis of rotation of 
the scanner). These figures are remarkable because the £, reconstruction has so 
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many added artifacts compared to the £5 reconstruction, although these operators 
are not very different (see section “Algorithms in Conical Tilt ET’). Why are the 
reconstructions so different? 

Reconstructions of real specimens from single axis tilt data show some of the 
same strengths and limitations (see, e.g., [80,83] and Oktem’s chapter in this book 
[71]). However, the added artifacts have different properties, and since the data are 
so noisy, other factors affect reconstructions. 


Synthetic-Aperture Radar Imaging 


In synthetic-aperture radar (SAR) imaging, a region of interest on the surface of the 
earth is illuminated by electromagnetic waves from an airborne platform such as a 
plane or satellite. For more detailed information on SAR imaging, including several 
open problems in SAR imaging, the reader is referred to [7,8] and to the chapter 
in this handbook by Cheney and Borden [9]. The backscattered waves are picked 
up at a receiver or receivers, and the goal is to reconstruct an image of the region 
based on such measurements. In monostatic SAR, the transmitter and receiver are 
located on the same platform. In bistatic SAR, the transmitter and receiver are on 
independently moving trajectories. 

While monostatic SAR imaging is the one that is widely used, bistatic SAR 
imaging offers several advantages. The receivers in comparison to transmitters are 
not active sources of electromagnetic radiation and hence are more difficult to 
detect if flown in an unsafe environment. Since the transmitter and receiver are 
at different points in space, bistatic SAR systems are more resistant to electronic 
countermeasures such as target shaping to reduce scattering in the direction of 
incident waves [48]. 

The reconstruction of the image based on the measurement of the backscattered 
waves is in general a hard problem. However, ignoring contributions of multiply 
backscattered waves linearizes the relation between the image to be recovered 
and the backscattered waves and is easier to analyze. Due to this reason, a 
linearizing approximation called the Born approximation that ignores contribution 
from multiply scattered waves is widely used in SAR image reconstruction. 


The Linearized Model in SAR Imaging 

Let yr(s) and yr(s) for s € (so, 5;) be the trajectories of the transmitter and receiver, 
respectively. The propagation of electromagnetic waves can be described by the 
scalar wave equation 


(4 = =i) E(x,t) = —P()8(x — yr(s)), S 


where c is the speed of electromagnetic waves in the medium, E(x,t) is each 
component of the electric field, and P(t) is the transmit waveform sent to the 
transmitter antenna. The wave speed c is spatially varying due to inhomogeneities 
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present in the medium, and one can assume that it is a perturbation of the constant 
background speed of propagation co of the form 


1 2 
AG) = oe + V(x). 


One assumes that V(x) only varies over a two-dimensional surface: the surface of 
the earth. Therefore, V can be represented as a function of the form 


V (x) = V(x)80(x3) 


where it will be assumed that the earth’s surface is represented by the x = (x1, x2) 
plane. The background Green’s function g is the solution of the following equation: 


(4 = =i) g(x, t) = —do(x)d0(Z). 
£0 


This is given by 


8(¢ = [xl feo) 


Ar ||>x|| 


gx, = (6) 
Now the incident field E due to the source s(x, t) = P(t)8(x — yr(s)) is 


E(x, t) 


II 


/ g(x —y,t —t)s(y, t)dydrt 


_ P= |x = yr(s)I| /co) 
4x ||x—yr(s)| 


Let E denote the total field of the medium, E = E™ + E*°, where E® is the 
scattered field. This can be written using the Lippmann-Schwinger equation: 


E*(z,t) = ic — x,t — 1)8; E(x, 1) V(x)dxde. 7) 


This equation is linearized by replacing the total field F on the right-hand side of 
the above equation by E"". This is known as the Born approximation. The linearized 


scattered wave field E}* (ypr(s), ¢) at the receiver location yp(s) is then 


Ex\in(Yr(S),t) = i; g(x — yp(s),t — 107 E™(x, t)V(x)dxdc. 


Substituting the expression for E into this equation and integrating, one obtains 
the following expression for the linearized scattered wave field: 
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Fig. 6 Reconstruction of a disk centered on the positive y-axis from integrals over ellipses (with 
constant distance between the foci) centered on the x-axis and with foci in [—3,3]. Notice that 
some boundaries of the disk are missing, and there is a copy of the disk below the axis. This 
was originally from the Tufts University Senior Honors thesis of Howard Levinson and published 
in [55, Reproduced with kind permission from Springer Science+Business Media: © Springer 
Verlag] 


lin 


Ex (yr(s),t) = i eZ REY) ACs. x, w)V(x)dxder, (8) 


where 


R(s, x) = |lyr(s) — xl] + [lx — yr@)I 


and 
A(s,x,@) = w* p(w) (42) |lyr(s) — x] lya(s) — x1). (9) 


where p is the Fourier transform of P. The function A includes terms that take into 
account the transmitted waveform and geometric spreading factors. The inverse of 
the norms appears in A due to the background Green’s function, (6). 

The reconstruction in Fig. 6 of a disk centered on the positive y-axis from 
integrals of it over ellipses with foci moving along the x-axis offset by a constant 
distance (which is simplified model of (8)) highlights some of the features in SAR 
image reconstruction. Some part of the boundary is not stably reconstructed, and an 
artifact of the true image appears as a reflection about the x-axis along with streak 
artifacts. Looking at the reconstructed image, one sees that at least visually, the 
created artifact is as strong as the true image. Microlocal analysis of the operators 
appearing in SAR imaging will make precise and justify all these observations. This 
will be addressed in section “SAR Imaging.” 
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General Observations 


In each reconstruction in this section, some object boundaries are visible and others 
are not. In fact, if one looks more carefully at the reconstructions, one can notice that 
in each case, the only feature boundaries that are clearly defined are those tangent 
to lines in the data set for the problem. Example | illustrates this in a naive way: 
one sees singularities in the Radon data exactly when the lines of integration are 
tangent to the boundary of the object. The goal of this chapter is to make the idea 
mathematically rigorous. 

The conical tilt ET reconstructions in section “Electron Microscope Tomography 
(ET) Over Arbitrary Curves” have artifacts if one uses a certain algorithm but 
apparently not when one uses another similar one. The reconstruction related to 
radar in Fig. 6 has an artifact that is a reflected image of the disk. 

In Sect. 4, deep mathematical ideas from microlocal analysis will be introduced 
to classify singularities and what certain operators do to them. In Sect.5 these 
microlocal ideas will be used to explain the visible and invisible singularities for 
limited data X-ray CT as well as the added singularities in ET and radar. 


3 Properties of Tomographic Transforms 


In this section, after introducing some functional analysis, basic properties of trans- 
forms in X-ray tomography and electron microscope tomography are presented. The 
microlocal properties of radar will be given later in section “SAR Imaging.” 


Function Spaces 


The open disk in R? centered at the origin and of radius r > 0 will be denoted D(r). 

The set C™(IR”) consists of all smooth functions on R”, that is, functions that 
are continuous along with their derivatives of all orders, and D(R”) is the set of 
smooth functions of compact support. Its dual space — the set of all continuous 
linear functionals on D(R") (given the weak-* topology) — is denoted D’(IR") and 
is called the set of distributions. If u is a locally integrable function, then u is a 
distribution with the standard definition 


(f= uf) = ff woo foes 


for f € D(R") since u(x) f(x) is an integrable function of compact support. 

If (2 is an open set in R”, then D(S2) is the set of smooth functions compactly 
supported in 92. Its dual space with the weak-* topology is denoted D’({2). 

The Schwartz space of rapidly decreasing functions is the set S(R”) of all 
smooth functions that decrease (along with all their derivatives) faster than any 
power of 1/ ||x|| at infinity. Its dual space, S’(IR"), is the set of all continuous linear 
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functionals on S(IR”) with the weak-* topology (convergence is pointwise: uz —> u 
in S’(R") if, for each f € S(R"), ux(f) > u(f)). They are called tempered 
distributions. Any function that is measurable and bounded above by some power 
of (1 + ||x||) is in S’(R”) since its product with any Schwartz function is integrable. 

A distribution u is supported in the closed set K if for all functions f € D(R”) 
with support disjoint from K, u( f) = 0. The support of u, supp(u), is the smallest 
closed set in which uw is supported. 


Example 2. The Dirac delta function at zero is an important distribution that is not 
a function. It is defined (50, f) = do(f) = f(0). Note that if f is supported away 
from the origin, then d9(f) = 0 since f(0) = 0. Therefore, the Dirac delta function 
has support {0}. 


Let €’(R") be the set of distributions that have compact support in R”. If 2 is an 
open set in R”, then €’(2) is the set of distributions with compact support contained 
in Q. For example, on the real line, 6 € €’ ((—1, 1)). 

If f ¢ L'(R"), then its Fourier transform and inverse are 


* 1 , 
FI®=fO= aon / ola ae 
. 1 ; (10) 
Ff) =F0)= Ga [OO e. 


The Fourier transform is linear and continuous from L! (R”) to the space of contin- 
uous functions that converge to zero at oo. Furthermore, F is an isomorphism on 
L?(R") and an isomorphism on S(R") and, therefore, on S’(IR"). More information 
about these topics can be found in [86], for example. 


Basic Properties of the Radon Line Transform 


In this section fundamental properties of the Radon line transform, ?,,, are derived, 
see [66]. This will provide a connection between the transforms and microlocal 
analysis in Sect. 4. 


Theorem 1 (General Projection Slice Theorem). Let f € L'(R?). Now leth € 
L®©(R) anda € S!. Then, 


CO 


/ F@RG-0) dx = i, Rif, pyh(p) dp. (11) 
xER2 Pp 


=—oo 


Proof, Let w € S'. First, note that the function x + f(x)h(x-q@) is in L'(R?) 
since / is bounded and measurable. For the same reason, the function 
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(p,t) > f(po + tor )h(p) 


is in L'(R?). Therefore, 


II 


/ f (x)h(x-@) dx iM [ f(pw + tw*)h(p) dt dp (12) 
xER2 Dp t 


=—00 


J Ri flo, pyh(p) dp (13) 
Dp 


=—oo 


where (12) holds by rotation invariance of the Lebesgue integral and then Fubini’s 
theorem and since p = w- (pw + tw+). The equality (13) holds by the definition 
of Ry. oO 


Let S(S! x R) be the set of smooth functions on S$! x R that decrease (along with 
all their derivatives) faster than any power of 1 / | p| at infinity uniformly in @, and let 
S’'(S! x R) be its dual. The partial Fourier transform is defined for g € L'(S! x R) 
as 


1 : 
Fyg(o,t) = ae 1 eT g(w,t) dt. (14) 
p 


Because the Fourier transform is an isomorphism on S(R), this transform and its 
inverse are defined and continuous on S’(S! x R). 
The Fourier Slice Theorem is an important corollary of Theorem 1. 


Theorem 2 (Fourier Slice Theorem). Let f € L'(R*). Then for (w,t) € S'xR, 


1 
Wire T) : 


F f(t) = 

To prove this theorem, one applies the General Projection Slice Theorem | to the 
function h(p) = e7!?". 

The Fourier Slice Theorem provides a proof that ? , is invertible on domain 
L!(R’) since F, p is invertible on domain L'(S!xR). Zalcman constructed a nonzero 
function that is integrable on every line in the plane and whose line transform is 
identically zero [107]. Of course, his function is not in L'(R?). 

This theorem also provides a proof of invertibility for the limited angle problem. 


Theorem 3 (Limited Angle Theorem). Let f € €'(R7) and let ® € (0, 2/2). If 
Rr f(@(¢), p) = 0 for g € (—®, ®) and all p, then f = 0. 

However, there are nonzero functions f € S(R?) with Ry f(@(@), p) = 0 for 
gy € (—-®, ®) and all p. 
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Proof. Let f € €’(R?) and assume Ry f(w(g), p) = 0 for y € (—®, ®) and all p. 
By the Fourier Slice Theorem, which is true for €’(R) [45], 


1 
V2 


and this expression is zero because Rx f(w(y), T) = 0 for such (g, T). This shows 
that F f is zero on the open cone 


F f (tw(g)) = F,Rif(o(y).t) =0 forge (-%,%),r ER (15) 


V = {ta(y) : 1 40,9 € (-@, G)}. 


Since f has compact support, Ff is real analytic, and so Ff must be zero 
everywhere since it is zero on the open set V. This shows f = 0. 
To prove the second part of the theorem, let f be any nonzero Schwartz function 


supported in the cone V and let f = F~! ( 7 i; Since ie is nonzero and in S(R’), 


so is f. Using (15) but starting with Ff = 0 in V, one sees that Rz f is zero in 
the limited angular range. Oo 


Another application of these theorems is the classical Range Theorem for this 
transform. 


Theorem 4 (Range Theorem [28,41]). Let g € S(S! x R). Then g is in the range 
of Ry, on domain S(R?*) if and only if 


1. g(@, p) = g(—@,—p) forall (w, p)< S'xR 


2. For eachm € {0,1,2,...}, g(w, p)p” dp is a polynomial in w € S' that 
peR 
is homogeneous of degree m. 


Proof (Sketch). The necessary part of the theorem follows by applying the General 
Projection Slice Theorem to h(p) = p” for m a nonnegative integer: 


Ri f(o, p)p" dp = / Fx\(r-a)" dx 
R x€R?2 


pe 


and after multiplying out (x-q@)’” in the coordinates of w, one sees that the right- 
hand integral is a polynomial in these coordinates homogeneous of degree m. The 
sufficiency part is much more difficult to prove. One uses the Fourier Slice Theorem 
to construct a function f satisfying F f(tw) = Tz pg(w,t). Since Fyg is 
smooth and rapidly decreasing in p, ¥ f is smooth away from the origin and rapidly 
decreasing in x. The subtle part of the proof in [41] is to show F f is smooth at the 
origin, and this is done using careful estimates on derivatives using the moment 
conditions, (2) of Theorem 4. Once that is known, one can conclude F f € S(R?) 
and so f € S(R’). Oo 
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The support theorem for R , is elegant and has motivated a large range of 
generalizations such as [5, 6,42, 53,57, 78]. 


Theorem 5 (Support Theorem [10, 28,41]). Let f be a distribution of compact 
support (or a function in S(R*)) and let r > 0. Assume Rf is zero for all lines 
that are disjoint from the disk D(r). Then supp(f) C D(r). 


This theorem implies that the exterior problem has a unique solution; in this 
case, D(r) is the excluded region. The proof is tangential to the main topics of this 
chapter, and it can be found in [10, 28, 41,43, 92]. 

Counterexamples to the support theorem exist for functions that do not decrease 
rapidly at oo (e.g., [43, 106] or the singular value decompositions in [73, 76]). 

A corollary of these theorems shows that exact reconstruction is impossible from 
ROI data where D(r) is the disk centered at the origin in R? and of radius r > 0. 


Theorem 6. Consider the ROI problem with region of interest the unit disk D(1). 
Letr € (1,00). Then there is a function f € D(D(r)) that is not identically zero in 
D(1) but for which Rf is zero for all lines that intersect D(1). 


Proof (Sketch). Let h(p) be a smooth nonzero nonnegative function supported in 
(1,7r) and let g(w, p) = h(|p|). Since g is independent of w, the moment conditions 
from the Range Theorem (Theorem 4), are trivially satisfied, so that theorem shows 
that there is a function f € S(R?) with R, f = g. By the support theorem, f is 
supported in the disk D(r). To show /f is nonzero in the ROI, D(1), one uses [10, 
p. 2725, equation (18)]. This is also proven in [66, p. 169, VI.4], and Natterer shows 
that such null functions do not oscillate much in the ROI. It will be shown in Sect. 5 
that null functions are smooth in the ROI, too. oO 


Continuity Results for the X-Ray Transform 


In this section some basic continuity theorems for R,, are presented. 

A simple proof shows that 2; is continuous from C.(D(M)) to C.(S17) where 
we define Sy = S! x [—M, M]. First, one uses uniform continuity of f to show 
Rx. f is a continuous function. Then, the proof that 7, is continuous is based on 
the estimate 


IR f(@, p)| < 7M? || f loo 


where || f||,, is the (essential) supremum norm of /. A stronger theorem has been 
proven by Helgason. 


Theorem 7 ([41]). Rz, : S(R*) > S(S! x R) is continuous. 
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The proof of the next theorem follows from the calculations in the proof of the 
General Projection Slice Theorem. 


Theorem 8. R, : L'(R?) > L!(S! x R) is continuous. 
Proof. By taking absolute values in (11) with A = 1 and then integrating with 
respect to w, one sees that ||f\lpiq@2 2 (27) [Ri flliusixey) and so Ry is 


continuous on L!, Oo 


Continuity results for Rz, in Sobolev spaces were given in [40, 45, 59] for 
functions of fixed compact support. 


Filtered Backprojection (FBP) for the X-Ray Transform 


To state the most commonly used inversion formula, filtered backprojection, one 
first defines the dual line transform. For g € L'(S! x R) and x € R’, 


Ri g(x) = ; g(@,x-@)do. (16) 


wéeS 
For each m € S', x € £(w,x-@), so Ri g(x) is the integral of g over all lines 
through x. The transform 7 is the formal dual to Rz in the sense that for f € 
S(R’) and g € S(S! x R), 
(Rif 8) 12(91xey = (ARLES) 12(@@% - 
Because R; : S(R*) > S(S! x R) is continuous, R¥ : S’(S! x R) > S’(R’) is 


weakly continuous. 
The Lambda operator is defined on functions g € S(S! x R) by 


Apg(, p) = Fz" (\t| (Fog, -))). (17) 


Theorem 9 (Filtered Backprojection (FBP) [66,85,87]). Let f € S(R*). Then, 
1 o 
f = DRIARif. (18) 


This formula is valid for f € €'(R?). 


Filtered backprojection is an efficient, fast reconstruction method that is easily 
implemented [67] by using an approximation to the operator A, that is convolution 
with a function (see, e.g., [66] or [87]). Note that FBP requires data over all lines 
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Fig. 7 FBP reconstructions of phantom consisting of three ellipses. The /eft reconstruction uses 
18 angles, the middle 36 angles, and the right one 180 angles 


through the object — it is not local: in order to find f(x), one needs data Rz, f over 
all lines in order to evaluate A,R, f (which involves a Fourier transform). 

To see the how sensitive FBP is to the number of the angles used, reconstructions 
are provided in Fig.7 using 18, 36, and 180 angles. One can see that using too 
few angles creates artifacts. An optimal choice of angles and values of p can be 
determined using sampling theory [15, 16, 66]. 


Proof (Proof of Theorem 9). Let f € S(R*). First, one writes the two-dimensional 
Fourier inversion formula in polar coordinates: 


1 ; 7 
= —_ ix + (tw) I 
FO) = sams Las deat fomsleldedo (19) 
1 eit(w* x) 
7 An iz fl Ven iM (FrRif) (@, t) dr dw (20) 
1 —_ 
thee bee (ApRif) (@,0-x) do = Ag REApRif(). (21) 
wes! 


The factor of 1/2 in front of the integral in (19) occurs because the integral is over 
t € Rrather than t € [0,c0). The Fourier Slice Theorem (Theorem 2) is used 
in (20), and the definitions of A, and of #7 are used in (21). The integrals above 
exist because f, Ff, and Rx f are all rapidly decreasing at infinity. 

The proof that the FBP formula is valid for f € €’(IR) will now be given. Since 
f € €'(R’), the Fourier Slice Theorem holds for f [45], and 


FyRi f(w,t) = V2nF f(t) 


is a smooth function that is polynomially increasing [86]. So, |t| -,Rz(@, T) is 
a polynomially increasing continuous function and therefore in S’(S! x R). Since 
the inverse Fourier transform, re maps S’ to S’, A,Rz_f is a distribution in 
S'(S! x R). By duality with S, Rf : S’(S' x R) > S’(R’), so RF A,Rif is 
defined for f € €’(IR”). Since the Fourier Slice Theorem holds for f [45], the FBP 
formula can be proved for f as is done above for S (see [26] more generally). O 
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Fig. 8 ROI reconstruction by 
Tufts undergraduate Stephen 
Bidwell from simulated data 
for the characteristic function 
of a circle using the operator 
Ly, given in (23) [4, ©Tufts 
University ] 


Limited Data Algorithms 


In limited data problems, some data are missing, and reconstruction methods are 
now presented for ROI CT, limited angle CT, and limited angle ROI CT. 


ROI Tomography 


Lambda tomography [17, 18,96] is an effective and easy-to-implement algorithm for 
ROI CT. The fundamental idea is to replace A, by —d*/dp* in the FBP formula. 
The relation between these two operators is that A = —d*/dp*, which will be 
justified in Example 9. This motivates the definition 


d2 
Lf = (-saR-s) : (22) 


dp? 


FBP is not local; to calculate A, f(w, p), one needs data over all lines to take 
the Fourier transform ¥,,. Therefore, one needs data over all lines through the object 
to calculate f using FBP (18). This means that FBP cannot be used on ROI data. 

The advantage of Lambda tomography over FBP is that £, is local in the 
following sense. To calculate £, f, one needs the values of (—d 2} dp’) Rif atall 
lines through x (since RF evaluated at x integrates over all such lines). Furthermore, 
(—d*/dp’) is a local operator, and to calculate (—d?/dp’) Rf at a line through 
x, one needs only data R, f over lines close to x. Therefore, one needs only data 
over all lines near x to calculate £, f. Therefore, £, can be used on ROI data. 
Although Lambda CT reconstructs £,f, not f itself, it shows boundaries very 
clearly (see Fig. 8 and [17]). 

Kennan Smith developed an improved local operator that shows contours of 
objects, not just boundaries. His idea was to add a positive multiple of R7Rz f 
to the reconstruction to get 
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1 a 
Lif = [oe ((-5 + 1) Rif) (23) 


for some jz > 0. Using (35), one sees that 


RE (UR f) (x) = (4 ; r) (x), oe 


lll 


so this factor adds contour to the reconstruction since the convolution with 2y/ ||x|| 
emphasizes the values of f near x. Lambda reconstructions look much like FBP 
reconstructions even though they are local. A discussion of how to choose jz to 
counteract a natural cupping effect of £, is given in [17]. The operator £, ,, is local 
for the same reasons as £, is, and it was used in the ROI reconstruction in Fig. 8. 

The ideas behind Lambda CT can be adapted to a range of limited data problems 
including limited angle tomography(e.g., [49, 56]), exterior tomography [79], and 
three-dimensional problems such as cone beam CT [2, 25,51, 103] and conical tilt 
electron microscopy [23]. Here is one such application. 


Limited Angle CT 


There are several algorithms for limited angle tomography (e.g., [3, 12,49, 61, 104]), 
and two will be presented that are generalizations of FBP and Lambda CT. The key 
to each is to use the limited angle backprojection operator that uses angles in an 
interval (—®, ®) with @ € (0, 2/2): 


@ 


RE img (X) = / _ (oly). x09) Mp. (25) 


The limited angle FBP and limited angle Lambda algorithms are 


2 


d 
RijtimApRif and Ri jim (- ie ) Rif (26) 


respectively. The objects in Fig.3 are reconstructed using this limited angle FBP 
algorithm. Limited angle Lambda CT is local, so it can be used for the limited angle 
ROI data in electron microscope tomography [80, 83]. 


Fan Beam and Cone Beam CT 
The parallel beam parameterization of lines in the plane used above is more 


convenient mathematically, but modern CT scanners use a single X-ray source that 
emits X-rays in a fan or cone beam. The source and detectors (on the other side of 
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the body) move around the body and quickly acquire data. This requires different 
parameterizations of lines. 

The fan beam parameterization is used if the X-rays are collimated to one 
reconstruction plane. Let C be the curve of sources in the plane, typically a circle 
surrounding the object, and let (w, 0) € C x S!. Then, 


L(o,0) = {w+ té:t >O} 


is the ray starting at w in direction 0, and the fan beam line transform is 


[ f(@ +t) dt. 
r=0 


In this case the analogs of the formulas proved above are a little more complicated. 
For example, a Lambda-type operator can be designed by taking the negative second 
derivative in 6 € S!. The other formulas are similar and one can find them in 
[66, 90]. 

In cone beam tomography, the source is collimated to illuminate a cone in space, 
and the source moves in a curve around the body along with the detectors. This 
scanner images a volume in the body, rather than a planar region as 7, and the 
fan beam transform does. However, the reconstruction formulas are more subtle 
[50], and one can understand these difficulties using microlocal analysis [25, 30]. 
Related issues come up in conical tilt ET as will be described in section “Microlocal 
Analysis of Conical Tilt Electron Microscope Tomography (ET).” 

These data acquisition methods have several advantages over parallel beam data 
acquisition. First, the scanners are simpler to build than the original CT scanners, 
which took data using the parallel geometry, and so a single X-ray source (or several 
collimated sources) and detector(s) were translated to get data over parallel lines in 
one direction, and then the source and detector were rotated to get lines for other 
directions. Second, they can acquire data more quickly than old style parallel beam 
scanners since the fan beam X-ray source and detector array move in a circle around 
the object. 

This is all discussed in Herman’s chapter in this book [44]. 


Algorithms in Conical Tilt ET 


Conical tilt ET [108] is a new data acquisition geometry in ET that has the potential 
to provide faster data acquisition as well as clearer reconstructions. The algorithms 
used for the conical tilt ET reconstructions will be given in section “Electron Micro- 
scope Tomography (ET) Over Arbitrary Curves.” This will lay the groundwork to 
understand why the reconstructions in that section from two very similar algorithms 
are so dramatically different. The model and mathematics are fully discussed in 
Oktem’s chapter in this book [71]. 
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First the notation is established. For w € S”, denote the plane through the origin 
perpendicular to w by 


wo? ={x ER? : x-w = 0}. (27) 
The tangent space to the sphere S 2 is 
T(S’) = {(@, x) : @ € S?,x €w?} 


since the plane w” is the tangent plane to S* at w. This gives the parallel beam 
parameterization of lines in space: for (w, x) € T(S7), the line 


L(@,x) ={x+tw:t eR}. 
If w is fixed, then the lines L(w, x) for x € w? are all parallel. As noted in Oktem’s 


chapter in this book [71], ET data are typically taken on acurve S C S*. This means 
the lines in the data set are parameterized by 


Ms = {(@,x) : wo E S,x Eo}. 


So, for f € L'(R*), the ET data of f for lines parallel to S can be modeled as the 
parallel beam transform 


Psflw.x) =f fle+ toa for (w,x) € Ms. 


Its dual transform is defined for functions g on Ms as 


Ps a(x) -| g(w,x —(x-w)o)do, 


We 


where dw is the arc length measure on S. This represents the integral of g over all 
lines through x. 

In this section, conical tilt ET is considered; an angle a € (0, 2/2) is chosen, and 
data are taken for angles on the latitude circle: 


Sq = {(sin(a) cos(@), sin(@) sin(y), cos(a))} : g € [0, 2z]}. 


Let Cy be the cone with vertex at the origin and opening angle a from the vertical 
axis: 


Cy = {tw : w € Sy}. 


Note that C, is the cone generated by Sy. 
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Here are the two algorithms for which reconstructions were given in sec- 
tion “Electron Microscope Tomography (ET) Over Arbitrary Curves.” The first 
operator is a generalization of one developed for cone beam CT by Louis and Maak 
[62]: 


Laf =Ps(—-As)Ps f. (28) 
where As is the Laplacian on the detector plane, w”. The second operator is defined 
Ls f = Ps(—-Ds)Ps f, (29) 
where Ds is the second derivative on the detector plane w” in the tangent direction 
to the curve S' at w (see Oktem’s chapter in this book [71]). The next theorem helps 


clarify these operators by writing them as convolution operators. 


Theorem 10. Let Ps be the conical tilt ET transform with angle a € (0, 1/2). Let 
f € €'(R’). Then 


pipsfafetr=[ IM, (30) 
yec, lly ll 
Laf =(-A)(f *1) (31) 
2. 
Ls f= (-4 + ese*(a) 75] fl (32) 
dz 


where I is the distribution defined for f € D(IR?) by 


I(f) = PLO) or 
yECy ma 


and dy is the surface area measure on the cone Cy. 

Equation (30) makes sense since P¢ integrates Ps f over all lines in the data set 
through x, and these are exactly the lines in the shifted cone x + Cy. The theorem 
implies that each of the operators is related to a simple convolution with a singular 


weighted integration over the cone Cy. 


Proof. The theorem is first proven for f € D(IR°), by calculating (30): 
PSPs fix) = f / f(x —(x-@)o + tw)dt da 
wéS JteR 


-| T(x + sw) ds da 
ow€S JteR 
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where the substitution s = t — (x - @) is used. Now, one converts this to an integral 
over Cy: 

* 1 

PsPs f(x) = f(x + sw@)— |s| dsdw 

weS JsER |s| 


ee y) 


dy 
yer, III 
since the measure on the cone Cy is dy = |s| ds dw where y = sa. 
To prove (31), one moves A inside the integral. Then one uses rotation invariance 
of A (to write A in coordinates: 


(s,t, p) (sm + ta! + pw xo’) 


where w’ is the unit vector tangent to S at w and in direction of increasing 9). 
Finally, one uses an integration by parts to show that P¢ intertwines A and As. To 


prove (32), one uses (31) and a calculation to show that (-4 + csc? (a) z=) and Ds 


are intertwined by P¢. 

The theorem is now proven for f € €’(R*). Since f has compact support, the 
convolution f « J is defined by [86, 6.37 Theorem]. Then, the proof is completed 
using the fact that the equalities are true for f ¢€ D(R*) and continuity of the 
operators (since Ps is a Fourier integral operator, which will be discussed in 
section “Microlocal Analysis of Conical Tilt Electron Microscope Tomography 
(ET)”). Oo 


4 Microlocal Analysis 


The reader has seen how limited data reconstructions can image different singular- 
ities (or add artifacts) depending on the type of data. This section will be devoted 
to developing the mathematics to understand those differences. The key point is to 
develop a precise definition of singularities and to understand what our tomographic 
transforms do to singularities. 


Singular Support and Wavefront Set 


Definition 1. Let 2 C R"” be an open set and let u € D’(2). The singular support 
of u, denoted by ssupp(u), is the complement in £2 of the largest open set on which 
uis C™® smooth. 


In other words, a point x9 € £2 is not in the singular support of u if u is smooth in a 
neighborhood of xo. Let us consider some examples. 
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<«—| > one 
<« | 
<« | 
f=0 <« | 
< 


Fig. 9 The function f = 1 in the interior of the square and f = 0 in the complement. The 
singular support, ssupp(f), is the boundary of the rectangle, and the wavefront set directions are 
shown in the figure 


Example 3. Consider the square S = [0,1]* in R*. Let f be the characteristic 
function 


Gee 1 if(%,y)eS; (33) 
; 0 otherwise. 


Then ssupp(u) is the boundary of the square because that is where uw is not 
smooth; see Fig. 9. 

Smoothness of a distribution u € €’(IR") is related to the rapid decay of its 
Fourier transform. Recall the following definition. 


Definition 2. A function f : R"” — C is rapidly decaying at infinity if for every 
N > 0, there is a Cy such that | f(x)| < Cy (1 + ||x||)~% for all x € R”. 


Theorem 11 ([(86]). A distribution u € E'(Q) is in C&°(Q2) if and only if its Fourier 
transform is rapidly decaying at infinity. 


This theorem implies that if a distribution u is not C® smooth, then there are 
nonzero frequency directions € such that the Fourier transform a does not satisfy 
the estimate of Theorem 11 in any conic neighborhood I" containing €. However, 
this is global information; it does not yet relate to singular support — the points 
where u is not smooth. To make this connection, one needs to consider directions 
near which a localized Fourier transform of u does not satisfy these estimates. This 
leads us to the concept of C™ wavefront set. 
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Definition 3. Let u be a distribution defined on an open set 2 C R”. One says that 
(xo, &) € 2 x R" \ 0 is not in the wavefront set of u, if there isa w € C2°(X) 
identically 1 near xo and an open cone I" containing & such that given any N, there 
is a Cy such that 


rulG)| < Cv + IID" forge Fr. 


On the other hand, the C™ wavefront set of a distribution u will be denoted by 


Remark I. To be more precise, one views elements of the wavefront set to be 
elements of the cotangent bundle T* 2 \ 0 using the notation 


Edx = &dx, +---+ &,dx, for & € R” \ 0. 


Through this, one can define wavefront sets for distributions on manifolds. 


Note that the cutoff function, w, in this definition is somewhat more restrictive 
than what is sometimes given (just that (xo) 4 0) but it is equivalent [47]. 


Theorem 12 ([(47]). Let u be a distribution defined on an open set 2 C R" and nx, 
denote the x-projection WF(u). Then 1 (WF(u)) = ssupp(u). 


Example 4. Consider the Dirac delta distribution 59 in R”. Then ssupp(59) = {0} 
because 59 is zero away from the origin and supported at the origin. So, by 
Theorem 12, x = 0 is the only point above in which there can be wavefront set. 
Furthermore, if y is a cutoff function at x9 = 0, then F(wd9) = 1/(2z)"/?, so 


WF(50) = {(0, &dx), & # O}. 


Example 5. Consider the function f given in Example 3. The wavefront set WF(/) 
consists of the nonzero normal directions at all the singular support points except the 
four corner points. At these corner points, all nonzero directions are in the wavefront 
set, as illustrated in Fig. 9. 

The proof is as follows. Consider first a non-corner point xo in the singular 
support. One can assume that this point is on the x-axis, x» = (a,0), where 
a € (0,1). Fix a direction €° = (&?, £9) with &? 4 0. One can show that the 
localized Fourier transform is rapidly decaying in a conic neighborhood of £°. To see 
this, one can find a narrow conic neighborhood I’ containing €° such |&| > c ||E|| 
for some c > 0 and all & € I” (here € = (&), &)). Let py € C®(R’) be a function 
of the form g(x1, x2) = @1(X1)@2(X2) that is identically 1 near x9. Without loss of 
generality, one can assume @ is even about a and @ is even about 0. Consider 
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co 


co 
a, 1 ix,* 1 —ix2& 
ef (&) = ae / em Heu(xid = fe 2825 (x2)dX . 
—oo 0 


Denote the left-hand integral in this expression by L(&) and the right-hand integral 
by R(&). Note that the R is bounded in & because the integrand is uniformly 
bounded and of compact support. Because g; is in S(R), L(&) is also in S(R), 
since it is the one-dimensional Fourier transform of g,. Therefore, L(&,) is rapidly 
decaying at infinity as a function of &. Since |&| > c |[&|| in I’, the function 
Eb} L(&,) decays rapidly at infinity for € in I”. Since the function R(&2) is bounded, 
one sees that of (€) decays rapidly in I”. This shows that the only possible vectors 
in WF(f) above xo = (a, 0) are vertical ones. 

Since f is not smooth at xo, at least one vertical vector at x9 must be in WF(/) 
by Theorem 11. Therefore, R(&) must not rapidly decay in either the positive 
direction (€; > 0) or the negative direction. Since @ is an even function in S(R), 
F (02)(&) = R(&) + R(—&) is rapidly decreasing at too, so R(&) must not be 
rapidly decaying for & > 0 and for & < 0 (since R(&) is not rapidly decaying in 
at least one direction and the sum is rapidly decaying in both positive and negative 
directions). Therefore, both vertical vectors are in WF(/) at xo. (In another proof, 
one shows R(&) = O(1/ |&|) by performing two integrations by parts on that 
integral.) 

Now it will be shown that all directions are in WF(f) above (0, 0). One can use 
symmetric cutoffs in x; and x2 at 0. Then 


1 
V2 


eo 8282 (2 (X2)dx2 


gf (&) = 


1 lo) 

ee e816 (x) )dxy. 
V2 | 

‘a 0 


The proof for R(&) above can be used to show that neither integral decays rapidly 
at infinity in this case. This shows that WF(¢/) consists of all directions at x9 = 0. 
The proofs at the other corners are similar. 


Example 6. If f is the characteristic function of a set, 2 with a smooth boundary, 
then WF(f ) is the conormal bundle: 


N*(Q) = {(x,&dx) : x € bd(), & is normal to bd() at x} 


This is suggested by Example 5 and it follows from the results in [47]. 

If f is a linear combination of characteristic functions of sets with smooth 
boundary, then WF(/) is the union of the conormal sets of the individual sets unless 
cancelation occurs along shared boundaries. 
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Pseudodifferential Operators 
To motivate the definition of these operators, consider the following example. 


Theorem 13. For f € S(R’), 


ae ae ae 1 re | 
RERiu(x) = fe ede = = 1 puorddvdé. 4 


Proof. Using a polar integration about x, one shows that 


2 
RIRLf = f*x—. (35) 
I|-ll 


Then, since F (1/ ||x|]) = 1/ |l&|| [43, Lemma 6.2, p. 238], one sees that 


R*Ryu=F'F (us 4 ) Fo} (2 = i) 
L = —. — —Uu 
" Il] IIE] 


using the fact that the Fourier transform of a convolution in R* (with the normaliza- 
tion of (10)) is the product of the Fourier transforms times 27. Writing F—! as an 
integral in the right-hand expression proves the theorem. EI 


It should be pointed out that the left-hand integral in (34) converges for f € 
S(R?), but the right-hand integral in (34) does not converge. However, one can do 
integrations by parts at infinity to make it converge for f € S(IR*) or f € €’(R’) 
as is done for pseudodifferential operators (e.g., [74]). 

With this as the model, consider operators with integral representation 


Pu(x) = / el—-E p(x, y, Eu y)dy db. (36) 


The study of the operator P is important in imaging for the following reasons: 


1. Assuming p satisfies certain estimates (see Definition 4), one can describe 
precisely the action of P on the singularities of u. 

2. If one has a procedure to invert or approximately invert the operator P by another 
operator Q having a similar integral representation as that of P, then by (a), 
one would have that the singularities of OPu are identical to those of u. Now 
through this approximate inversion process, one has a procedure to recover the 
singularities of u. 
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An operator P of the form (36) with p satisfying certain estimates is called a 
pseudodifferential operator (YDO) [33, 47, 89, 93, 94]. This will be given below 
(see Definition 5). 

In order to motivate the appropriate conditions and estimates that p should 
satisfy, let us look at the following simple example: 

Consider a linear partial differential operator of the form 


PED) =->~ a(@)Ds. (37) 
|v|<m 
Here v = (11,-+: , vz) is a multi-index and 
QU! oun 
DY = (ij) Mite) sis . 
ee, Ox," Oxn" 


For simplicity let u be a compactly supported function. Applying the Fourier 
transform, 


Diu(é) = [ete sptucae. (38) 


1 
(27)"/2 


Integrating by parts |v| times, one obtains 


D'u(é) = €°i(E). (39) 
With this one has 
1 Pate 
P(x, Dus) = ae f el” p(x, ul y)aya (40) 
where 


p(x.&) =D ay(x)é”. 


The function p(x, &) is called the symbol of the partial differential operator (PDO), 
P(x, D). 

The function p(x, &) satisfies the following property: Let a = (a ,--: ,a@,) be 
a multi-index. Then df p(x, &) lowers the degree with respect to & of the resulting 


function by |a|, whereas db p(x, &) where 6 = (1,--- , Bn) is a multi-index does 
not alter the degree of homogeneity with respect to & of the resulting function. 
More precisely one has the following estimate: 
Let a and 6 be any multi-indices. For x in a bounded subset of R”, there is a 
constant C such that 
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agae p(x.8)| < CU + lle. (41) 


Here m is the order of the PDO, P(x, D). In order to get this inequality, first rewrite 
the terms of p(x, &) by combining terms of the same homogeneous degree with 
respect to the & variable. 

Differentiate €” a = (a,--- ,a@,) times with respect to &,--- ,&,, where the 
number of times one differentiates &” with respect to each particular & is a. One 
sees that this reduces the degree of homogeneity of &” by |a| and the highest-order 
terms dominate. On a bounded subset of R”, all derivatives of the a, are bounded. 
This gives the estimate (41). 

Now one can generalize the class of operators that have Fourier integral 
representations of the form (40) by admitting a larger class of functions p(x, &) to 
be symbols. Consider those functions p that satisfy the estimate as in (41) and that 
behave like polynomials or the inverse of polynomials in & as ||&|| — oo. In other 
words, one wants p(x, &) to grow or decay in powers of ||&||, and differentiation with 
respect to € lowers the order of growth or raises the order of decay. Furthermore, in 
order to include R7R, in the class of operators considered (see Example 13), one 
can allow some latitude at & = 0. 

In the interest of flexibility, one can also let the function p depend on x, y, and 
&. Such functions will be denoted as amplitudes. 


Definition 4 ((33, 47, 89, 93,94]). Let X C R” be an open subset. An amplitude of 
order m is a function that satisfies the following properties: 


1. p(x, y.) € C°(X x X xR" \ {0}), 
2. For every compact set K C X and for multi-index a, f, y, 
(a) There is a constant C = C(K,a, B, y) such that 


De DE DY p(x, y,&)| < C+ |E ||)! for x and y in K and ||é|| > 1, and 


(b) p(x, y, €) is locally integrable for x and y in K and ||&|| < 1. 


It is important to note that in Definition 4, p need not be a polynomial in € and m 
can be any real number. The local integrability condition can be relaxed if p is a 
sum of homogeneous terms in & [74]. 

Now pseudodifferential operators are defined. 


Definition 5. Let X C R" be an open subset. A pseudodifferential operator (/DO) 
is an operator of the form 


I 
(2x)" 


Pu(x) = i elO—E p(x, y, Eu(x)dyde, 


where p(x, y, €) is a function that satisfies the properties of Definition 4. 
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The operator P has order m if its amplitude is of order m, and P is elliptic of 
order m if for each compact set K C £2, there is a constant Cx > O such that for x 
and y in K and ||&|| > Cx 


Ip(x, y.€)| = Cg t+ Ell)”. (42) 


The next theorem highlights two fundamental properties of YDOs. 


Theorem 14 (Pseudolocal Property [94]). Jf P is a WDO, then P satisfies the 
pseudolocal property: 


ssupp(Pu) C ssupp(u) and WF(Pu) C WF(u). 


If, in addition, P is elliptic, then 


ssupp(Pu) = ssupp(u) and WF(Pu) = WF(u). 


Note that, although YDOs can spread out the support of the function u, they do 
not spread out its singular support or wavefront set. Elliptic YDOs preserve singular 
support and wavefront set. 

Equation (34) shows that the composition Rj R, is a pseudodifferential operator 
since its symbol (47r)/ ||&|| satisfies the conditions in Definition 4. Furthermore, 
because the symbol satisfies the ellipticity estimate (42), R7 FR is an elliptic YDO 
of order —1. 


Example 7. The powers of d/dp and A as WDOs. 

According to (40), the symbol of —d*/dp? is |t|° where t is the dual variable to 
p. So, the symbol of ,/—d?/dp? can be considered to be |r|. The rationale is that if 
one calculates 


a* - = = 
gpl FUP FS HFEF IFS) 
d? d? 
= ap oy apt 


Ap = V~d?/dp?. (44) 


(43) 


This justifies why 


Since 


-A=D?,+---+D?, 
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its symbol is WlE||°. Since the symbol of —A is ||&||?, one can easily find symbols 
of powers of the Laplacian. For example, ~—A has symbol ||&|| and Fourier 
representation 


1 iG) 2 
Vu = / el—E HET u(y)dydé 


and (—A)~"’” has symbol 1/ |lé|| 


1 F 1 
Cay u= ao / eS nv dd 


Now, as a calculation in distributions using Fourier transforms (similar to (43)), one 
sees that (—A)-” > 6 V/A is the identity map. 

Note that /—d?/dp2, /—A, (—A)~'”” are all elliptic YDOs since the corre- 
sponding symbols satisfy the ellipticity estimate (42). 


Example 8. Example 7 allows one to justify why the operator A, is really 
J —d2/dp?. So, the FBP inversion formula (18) can be written 


f= Ri (V=PTaPRS) (45) 
JT 


Equation (34) shows that RFR, = 42 (—A)~'/ 2 Using the observation at the 
end of Example 7, one obtains a different version of the filtered backprojection 
inversion formula for R_: 


1 
f= iY ARIAS (46) 


These calculations can be justified for distributions of compact support [43, 45]. 


Example 9. Now consider the Lambda operators given in (22) and (23). To get the 
Lambda operator, £,, from the FBP operator, one replaces the ,/—d?/dp? in (45) 
by its square, —d*/dp?. 

Here is another way to understand Lambda tomography. By evaluating another 
./—A in (46), one sees that 


ee 1 if @ 
ba a a hc Me ia (-5R) 


where the second equality holds because R¥ intertwines —A and —d?/dp? (this 
is proven using an argument similar to the intertwining argument in the proof of 
Theorem 10), Because R7R, is an elliptic YDO with symbol 477/ ||&||, the symbol 
of Ly, is 
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a 


[Is] + il 


and it is elliptic of order one. Therefore, £,,, and £, (corresponding to = 0) are 
both elliptic WDOs. 


The operator £,,,, does not reconstruct f but ((-A)"/ - 


+ 1) jf. The natural 


question then is how different is this from f ? It has just been established that £, and 
Lx, are elliptic YDOs. Therefore, by Theorem 14 this means that these operators 
recover ssupp(f) and WF(/). The reconstructions in Fig. 8 and others in [17, 18] 
show that it is effective in practice. 


Fourier Integral Operators 

In the analysis thus far, one studied the composition of a generalized Radon trans- 
form with its adjoint. This composed operator, as was shown, is a pseudodifferential 
operator. Theorem 14 shows us how WDOs act on singularities and wavefront sets. 
In this section, more general operators will be considered, and how they change 


wavefront sets will be described. 


Example 10. Consider R, in a special Fourier representation. 


Ri fOr?) = ors | _ eM Fy (Rif) (0,2) de 


/ PF ae (47) 
TER 


= / i eile—x-wyyr ID f(x)dxdrt. 
TER JxER2 20 


The last expression in (47) looks like a WDO except that the t and x integrals are 
over different sets and the exponent is not the one for YDOs. 


In many applications, it might be necessary to understand the properties of the 
Radon transform directly, rather than the composition with its adjoint. In the last 
example one saw that the Radon transform 7, had an integral representation of the 
form 


Pulx) = i ei#®) n(x, y, Eyu(y)dyd€. (48) 


The important differences between the operator P in (48) and a WDO are the 
following: 
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¢ The functions Pu and u, in general, are functions on different sets X and Y, 
respectively. The spaces X and Y can be of different dimensions as well. 

¢ The phase function @ in (48) is more general than that of a YDO, but it shares 
similar features. See Definition 6. 

¢ The dimension of the frequency variable € can be different from that of the spaces 
X and Y, unlike as in the case of a WDO. 


Another simple example where an integral representation of (48) arises is when 
one uses Fourier transform techniques to determine the solution to a constant 
coefficient wave equation: 


(0? —Ay)u=0, u(x,0)=0, d,u(x,0) = g. (49) 


Now by taking Fourier transform in the space variable, one has the following integral 
representation for the solution to the wave equation: 


__ i i(x—y)-e-+ene 1 
se oa ° Elbiaees 


; 1 
— f eter HEL a(ypdyat. 


Note that the phase functions in the above solution are $4. (x, y,€) = (x —y)-E+ 


t {lél- 


If f : R” — R, then the notation 


0 ) 
dx f(x) = shan, eet oh ax, (50) 


will be used for the differential of f with respect to x. If g is a function of (9, p), 
then 


dg dg 
do.p8(Y, Pp) = ad + ap? 


will denote the differential of g with respect to the variables (¢g, p). 


Definition 6. Let X C R” and Y C R" be open subsets. A real-valued function 
db € C~(X x Y x RN \ {0}) is called a phase function if 


1. @ is positive homogeneous of degree | in &. That is, (x, y,r&) = rd(x, y,§) 
for all r > 0. 
2. (Ax, def) and (4,¢, def) do not vanish for all (x, y,€) € X x Y x R% \ {0}. 
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The phase function ¢ is called nondegenerate if on the zero-set d¢¢ = 0, the set of 
vectors {0x,.¢ a 1 < j < N} is linearly independent. 
y€ OE; 


Definition 7. A Fourier integral operator (FIO) P is defined as 


Pu(x) = / elf 5) n(x, y, E)u(y)dydé, 


where ¢ is a nondegenerate phase function, and the amplitude p(x, y,&) € C°(X x 
Y x RY) satisfies the following estimate: for every compact set K C X x Y and for 
every multi-index a, B, y, there is a constant C = C(K, a, 6, y) such that 


| Dg DEDY p(x, y,8)| < C(1 + /é||)""! for all x, y € K and for all  € R™. 


Finally, one has two important sets associated with this FIO. 
The auxiliary manifold 


Xp = {(x, y,€) € X x ¥ x (RY \ 0) : OG(x, y, &) = 0} (51) 


and the canonical relation 


C = {(x, G(x, , &) ¥, Oy (x,y, &)) : (x,y, £) € Ze}. (52) 


Note that since the phase function ¢ is nondegenerate, the sets Xy and C have 
smooth manifold structures. 

One can also include the local integrability condition 2b of Definition 4 for the 
amplitude of FIOs. 

Note that Rz satisfies these conditions with phase function ¢(@, p,x,T) = (p— 
x-+q)t and amplitude p(x, y,t) = 1/(27), so Rx is an FIO. Guillemin originally 
proved that a broad range of Radon transforms are FIOs [35, 36, 38]. The operator 
Ry will be studied more carefully in the next section. 

Every WDO is an FIO with phase function (x, y,&) = (x — y)-&. However, 
R_ is an FIO that is not a WDO since its phase function is not of that form. 


Definition 8. Let C Cc T*X x T*Y,C C T*Y x T*X, and A CT*Y. Define 
CoA= {(x, €dx) : d(y, ndy) € A with (x, dx; y, ndy) € C} 


CoC ={(5, ids: y,ndy) : 3 (x, Edx) with (5, Ag; x, Edx) € E, 


and (x, dx; y, ndy) € Ch. 
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The key theorem on what FIOs do to wavefront sets is the following result. 


Theorem 15 ([47]). Let P be an FIO and let C be the associated canonical 
relation. Then 


WF(Pu) C C o WF(u). 


Example 11. Yn this example, the canonical relation of any YDO will be calculated. 
First note that since @ = (x — y)-&, 


Ix@ = Edx Ip =—Edy deh = (x — y)dé. 


Therefore, 34 = {(x, y,&) : x — y = 0}. Now, use (52) and these calculations to 
see that the canonical relation for YDOs is 


C = {(x, Edx; x, dx) : & £0} 


which is the diagonal in (7*(R”) \ 0) and which will be denoted by A. Using 
Theorem 15, one sees that if P isa WDO, WF(Pu) C WF(u). 


This section will be concluded with a key theorem on the wavefront set of the 
composition of two FIOs known as the Hérmander-Sato Lemma. 


Theorem 16 (H6rmander-Sato Lemma [47]). Let P, and P2 be two FIOs with 
canonical relations C, and C2, respectively. Assume P; © P2 is defined for 


distributions of compact support, and let u be a distribution of compact support. 
Then 


WF((P) 0 P2)u) C (Ci ° C2) 0 WF(u). 


From an imaging point of view, the operator that is studied is the image recon- 
struction operator P*P where P* is the adjoint of the FIO P. Using Hérmander- 
Sato Lemma, one can study the wavefront set of the image reconstruction operator. 
For this one requires the canonical relation of the adjoint P* [46] which is given by 


C' = {(y, ndy; x, &dx) : (x, dx; y, ndy) € C}. 
Now from Hérmander-Sato Lemma, when the composition is defined, one has 


WF(P*P f) C (CoC) 0 WE(f). 
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5 Applications to Tomography 


In this section microlocal analysis will be used to explain strengths and limitations 
of the reconstructions from Sect. 2. 


Microlocal Analysis in X-Ray CT 


As seen in section “X-Ray Tomography (CT) and Limited Data Problems,” recon- 
structions from different limited data problems have different strengths and weak- 
nesses. The information in the last section will now be used to understand the 
microlocal properties of R. 

To make the cotangent space calculations simpler, the following coordinates will 
be used on S! x R: (g, p) & ((¢@), p) where w(~) = (cos(g), sin(g)) and 
o+(¢) = wo(g + 2/2). Thus, functions on S! x R will now be written in terms 


of (g, Pp). 


Theorem 17. The Radon transform Ry is an elliptic FIO associated with the 
canonical relation 


Cr = { (9, p.0(—x-@* (p)dg + dp): x, 2a()dx) 
(53) 
: (¢, p) € [0,22] x R,x € R’,a £0, x-w(y) = | 


For f € E'(R’), 


WF(R, f) = Ci o WF(f). (54) 


7) 


Furthermore, Cj, ° Cy, = A is the diagonal in (T*(R?) \ 0) 


Proof. In Example 10, it was shown that 7, is an FIO associated to the phase 
function 


P(Y, P.X,T) = tT(p—xX-o(9)). 


To calculate the canonical relation for R,, the general methods outlined in 
section “Fourier Integral Operators” (see also [47, p. 165] or [95, (6.1) p. 462]) 
will be followed. First, the differentials of @ will be calculated: 


Ix = —tH(y)dx, Ivy py = t (—x-w+(y)dy + dp) 


(55) 
dep = (p—xX-w(e))dt. 
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Note that the conditions for ¢ to be a nondegenerate phase function [95, (2.2)— 
(2.4), p. 315] hold because 0, and d(,p)¢ are not zero for t # 0. Therefore, 
Ry is a Fourier integral operator. Rz has order —1/2 because its symbol 1/27 is 
homogeneous of degree zero, 2 = dim R* = dim Y, and a is one dimensional (see 
[95, p. 462 under (6.3)]). Since the symbol, 1/27, is homogeneous and nowhere 
zero, R is elliptic (see [46]). 

The auxiliary manifold Xy is 


Zp = {(Y, p. x,t) € ([0, 27] x R) x R* x (R\0) : p—x-o(y) =0}. (56) 
The canonical relation, C,, associated to R,, is defined by the map 


Lp > (X., PT) > (9, (XO), yp); x, Ig) - 


One uses this and a calculation to justify the expression (53). 

To show C; o C, = A, one starts with (x, &dx) € T* (IR?) and then follows 
through the calculation of Cj; o Cy using (53): First, choose g € [0,27] such that 
— = aw(g) for some a > 0. Then, there are two vectors associated to (x, &dx) in 
CL, 


Ai = (9, x-@(y); a(—x-@* (y)dy + dp), 
Ar = (9+ 7,x-0(9 + 2); —a(—x-ot(o + n)dg + dp)). 


Under Cj, 4, is associated with ((x -w(9))@(y) + x-ot(@), aw(y)dx) and this 
is exactly (x, €dx), and there is no other vector in T*(IR*) associated with A,. Ina 
similar way, one shows that the only vector in 7* (IR*) associated with A> is (x, dx). 
Therefore, 


Cho Cy = {(x, €dx; x, Edx) : (x, &dx) € T*(R’)\ 0} = A. 


Because the symbol of R ,, 1/27, is nowhere zero and homogeneous, Ry, 
is elliptic [95]. Furthermore, because Cz is a local canonical graph (e.g., [75] 
and can be seen from the calculation in the last paragraph), a stronger version 
of the Hérmander-Sato Lemma for elliptic operators [95] can be used to show 
WF(Rz f) = Cr o WF(S). Oo 


Note that the fact C} o C, = A implies that WF (R7 Ri (f)) C WF(S), by the 
Ho6rmander-Sato Lemma (Theorem 16). This and the composition calculus of FIO 
[46, Theorem 4.2.2] provide another proof that R7R , is a WDO. 

This theorem has the following important corollaries. 


Corollary 1 (Propagation of Singularities for R,). Let f € €'(R’). 
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1. Let (xo, &9dx) € T* (IR?) \ 0 and let go be chosen so that & = aw(go) for some 
a #0. If (xo, &odx) € WF(S), then 


(0, Xo - @(Yo); @(—x9 -@*(Yo)dy + dp)) € WF(Rz f). 


2. Let (go, po) € [0,27] x Rand A € R. Assume (0, po;a(—Adg + dp)) € 
WE(R_ f). Then, (xo, &dx) € WF(f) where x9 = pow(vo) + Aw+(go) and 
&) = aw(%o). 


This provides the paradigm: 

Ry detects singularities of f perpendicular to the line of integration (“visible” 
singularities), but singularities of f in other (“invisible”) directions do not create 
singularities of Rf near the line of integration. 


Remark 2. The paradigm has implications for limited data tomography. A wave- 
front direction (xo, &dx) € WF(/) will be visible from limited Radon data if and 
only if the line through xo perpendicular to & is in the data set. 


Proof (Proof of Corollary 1). Using (54), one sees that 
(xo, §odx) € WF(f) if and only if Cz o {(xo, &odx)} € WF(RzS). 


Part (a) follows from the forward implication of this equivalence and part (b) follows 
from the reverse implication using the expression for C_, (53). 

Part (a) implies that 7, detects singularities perpendicular to the line being 
integrated over, since w(g) is perpendicular to the line L(w(q@o), x0: @(Go)). 
Part (b) implies that if a singularity is visible in Rz f at (go, po), it must come from 
a point on L(w(¢o), Po) and in a direction perpendicular to this line. This explains 
the paradigm in the theorem. Oo 


Corollary 2 (Propagation of Singularities for Reconstruction Operators). Let 

f €€'(R’). 

(a) Let L be either the FBP (see (18)), Lambda (Ly, (22)), or Lambda + contour 
(Ly yw» (23)) operator. Let f € E'(IR*). Then, WF(f) = WF(L Sf). 


(b) Let RF jim be the limited angle backprojection operator in (25) for angles a < 
y <b (where b—a <x). Let 


V = {(x,&dx) : € = aay), € (a,b)} . 


Tf Lim is any of the operators 


ies a 
RiampRis Rism (Sat). Rim ((—aa +) Re) 
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then 
WF(Lim f) AVY = WF(f) OY. 


Note that V is the set of visible singularities for the limited angle transform. 


Proof. Part (a) follows from the fact that both £, and £,,,, are elliptic, as noted in 
Example 9 and the equalities stated in Theorem 14. 

Part (b) follows from the fact that when one cuts off angles, one can see only 
wavefront parallel to the angles in the data set, that is, the visible directions in V. A 
complete proof of this result is given in [26]. oO 


Remark 3. The paradigms in Corollaries 1 and 2 have especially simple interpre- 
tations if f is the sum of characteristic functions of sets with smooth boundaries. 
The tangent line to any point on the boundary of a region is normal to the wavefront 
direction of f at that point (since the wavefront set at that point is normal to the 
boundary, as noted in Example 6). 

So, a boundary at x (with conormal (x, &dx)) will be visible from the data Ry, f 
near (w(q@), p) if (x, €) is normal to the line of integration. This can be stated more 
simply as follows: the boundary at x is tangent to a line in the data set. 

Finally, note that Example | provides a simple case for which Rz, f is not smooth 
when the line L(g, p) is tangent to the boundary of supp(/). The paradigm shows 
this principle is true generally. 


Limited Data X-Ray CT 


Each of the limited data problems discussed in Sect. 2 will now be examined in light 
of the corollaries and paradigm of the last section. 


Exterior X-Ray CT Data 


In the reconstruction in Fig.2, the boundaries tangent to lines in the data set are 
clearer and less fuzzy than the one not tangent to lines in the data set. The paradigm 
in Corollary | and Remarks 2 and 3 explains this perfectly. When a line in the data 
set is tangent to a boundary, then the boundary is visible in the reconstruction. For 
the exterior reconstruction in Fig. 2, this is true for the inside boundary of the disk at 
about eight o’clock on the circle (lower left), even though that boundary is imaged 
by only a few lines in the exterior data set. If the line tangent to the boundary at x is 
not in the data set, then the boundary is fuzzier, as can be seen in that figure. 

This is reflected in Quinto’s algorithm [77] in the following way. That algorithm 
expands the reconstruction in a polar Fourier series 
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f(ro@) = >> fe, 


eZ 


where f;(r) is approximated by a polynomial which is calculated using quadrature. 
The calculation of f; could be done stably only up to about |£| = 25. However, for 
each £ for which this could be done, the recovery of f;(r) is very accurate and could 
be done up to a polynomial of order about 100. Thus, the algorithm has good radial 
resolution but bad resolution in the polar direction. However, this is, at least in part, 
a limitation of the problem, not just the algorithm. The singularities in Fig. 2 that 
are smoothed by the algorithm are intrinsically difficult to reconstruct. 


Limited Angle Data 


In Fig. 3, data are given over lines L(w(@), p) for g € [—2/4, 7/4], and the only 
boundaries that are visible are exactly those normal to such lines. This reflects 
Corollary 1 and Remarks 2 and 3. The algorithm used is FBP but with a limited 
angle backprojection (see (25) and the discussion below it) between —2/4 and 
@ = 2/4. In this case, Corollary 2 implies that the only singularities of f that 
will be visible in Lim f (where Lijim is given in that corollary) are those in the cone 
VY = {ra(y) : 9 € (—-2/4, 7/4), r A O}. 

However, there is also a marked streak along the lines with angle g = +2:/4 that 
are tangent to the boundary of the region. Frikel and Quinto have recently explained 
this using microlocal analysis [26], see also [49]. They also explained that in order 
to decrease those streaks, one needs to make the backprojection operator a standard 
FIO by including a smooth cutoff function y supported in (—7/4, 2/4) that is equal 
to 1 on most of that interval: 


/4 
Rise) = [ 80M OA. 
g=—-n 


Region of Interest (ROI) Data 


The reconstruction in Fig. 8 is from Lambda CT, and all singularities of the circle 
are visible. This is expected because of Corollary 2 Part (a), the Lambda operator 
preserves wavefront set. Recall that £,(/)(x) determined by data R, f for lines 
near x. This means that the wavefront of f above x (which is the same as the 
wavefront set of £,(f) above x), can be determined by local data for lines near x. 
This is why all singularities are visible in the Lambda reconstruction in that figure. 
This also means that any null function for the interior problem must be smooth 
in the ROI since its Radon data are zero (e.g., smooth) for lines meeting the ROI. 
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Microlocal Analysis of Conical Tilt Electron Microscope Tomography 
(ET) 


Using the notation of section “Algorithms in Conical Tilt ET,” let 


Sq = {(sin(a) cos(¢), sin(@) sin(g), cos(@)) : g € [0, 27]} 
Cy = RS, . 


Using Theorem 10, one writes the operators £, (see (28)) and £5 (see (29)) using 
the convolution 


1 
f * I(x) = JOS y)—y 
y€Cy Ill 


where dy is the surface area measure on the cone C,. In particular the conclusion 
of that theorem is 


PsPsf =f *I 
Laf =(-A)(f* I) 


Lsf = (-4 + ese") ) fx. 
dz3 


As one can see from the reconstruction in the x —y plane, Fig. 4, there are circular 
artifacts. In the reconstruction in the vertical plane x = —y, Fig. 5, there are streak 
artifacts coming off from each of the balls at a 45° angle. 

These artifacts can be understood intuitively as follows. Let f be the character- 
istic function of a ball B. Then, f * J(x) integrates f over the cone x + Cy/4. 
When this cone is tangent to B at a point besides x, it is normal to a singularity 
of f at this point. This singularity will cause a singularity of f * J at x. The 
reason is, when x is perturbed, the cone moves in and out of B, so the integral 
changes from 0 (when the cone is disjoint from B) to nonzero values as the cone 
intersects B. A calculation will convince one that the singularity is a discontinuity 
in the first derivative. Because £, is a derivative of f « J, that singularity is 
accentuated in Ly f at x. One can see this phenomenon from the artifacts in Ly 
reconstructions in section “Electron Microscope Tomography (ET) Over Arbitrary 
Curves.” In the x — y plane, the artifacts are circular shadows from the disks outside 
of the plane; in the y = —x plane, the artifacts follow along the generating lines 
of the cone. However, this is not a rigorous explanation since it does not apply to 
arbitrary functions. It also does not explain why the £, reconstruction has apparent 
singularities, but the £5 reconstruction seems not to. 

The next theorem explains this using microlocal analysis. To state the theorem, 
recall the definition of conormal bundle of a submanifold B C R? as 
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N*(B) = {(y, €dx) : y € B,& is normal to B at y} . 


Theorem 18 ([23]). The conical tilt ET operator Ps is an elliptic Fourier integral 
operator. Let f € €'(R?) and let a € (0, 1/2). Let L be either Ly or Ls. Then, 


WE(L(f)) C (WE(f) NV) U A(f) 
where 
V=R>x {ne R?\0: Jo € So, n-w = 0} 


represents the set of possible visible singularities. The added artifacts come from 
A(f) = {@, &dx) : A(y, Edx) € N*(x + Ca) 0 WF(f)} 


Furthermore, the added artifacts in A(f ) are more pronounced (stronger in Sobolev 


scale) in Lyf thanin Ls f. 


Here V is the set of visible singularities, those from f that should appear in £/f. 
The set A(/) consists of added artifacts, those caused by wavefront of f normal to 
x+C,, at points besides x. In Fig. 4, the added artifacts in the 2, reconstruction are 
exactly those that come from cones x + C, that are tangent to boundaries of disks 
in supp(/), and this gives the same conclusions as the heuristic description given 
above Theorem 18. 

The final statement in the theorem follows from the fact that £, is in a class of 
singular FIOs (so-called IP" classes studied in [31,32,39,64]) and Ls is in a better 
behaved class. These classes of singular FIOs also come up in radar imaging, as 
discussed in the next section. 

The set of visible singularities V is larger for conical tilt ET than for the standard 
data acquisition geometry, single axis tilt ET, and this is one reason to take conical 
tilt data, even though acquiring data can be technically more difficult. 

The proof of this theorem uses the Hérmander-Sato Lemma (Theorem 16). 
One first calculates the canonical relation, C, and then one calculates C‘ o C and 
shows C' o C has two parts. One part generates V, the set of visible singularities, 
singularities of f that will be visible in £4 f. The second part generates the set of 
added artifacts, A(/). 

Deep results in [31] are used to explain why Ls suppresses the added singu- 
larities better than £,. This can be explained intuitively as follows. If an added 
singularity at x comes from a point (y,&dx) € N*(x + Cy) N WF(f), Ds 
takes a derivative corresponding to a derivative in x that is normal to & (so 
it does not increase the order of this singularity) and A takes a derivative in 
the direction of this singularity (so it does increase the order of this singular- 


ity). 
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This conical tilt transform is an admissible Radon transform [27]. The microlocal 
analysis of such transforms was first studied in [36], and the microlocal properties 
of these transforms were completely analyzed in a very general setting in [30] 
including general results for backprojection that are related to Theorem 18. 

Quinto and Rullgard have proven similar results for a curvilinear Radon trans- 
form for which the more effective differential operator (analogous to Ds) decreases 
the strength of artifacts only locally [81]. 


SAR Imaging 


In this section, some recent results on SAR imaging will be described from the 
viewpoint of microlocal analysis. The notation is from section “Synthetic-Aperture 
Radar Imaging.” Recall that the forward operator under consideration is 


PV(s,t) = ye io(t—(\|x—yr (9) + Il yR(3)I)/¢0) A(s, t,x, @)V(x)dxdo. (57) 


In this section w € R. The canonical relation wherever it is well defined is given as 
follows. This is an important set that relates the singularities of the object V to that 
of the data PV: 


c= }s-o( (FEA nto + aH yh) asa} 


x —yr(s) x — yr(s) ) 
»X2,—- d 58 
mee 0 (Tt peal ‘ oe 


> Cot = [|x — yr(s)|| + llx — yr(s)||.o F f 


Note that (s, x, @) is a global parametrization for C. Let us denote 
Y = {(s,t) € (0, 00) x (0, oo)} 


and {x,,x2} space as X. One is interested in studying the imaging operator P*P. 
The standard composition calculus of FIOs, the so-called transverse intersection 
calculus of Hérmander [46], and the clean intersection calculus of Duistermaat and 
Guillemin [14,34] and Weinstein [98] do not apply in general in these situations. 
Therefore, one approach to understanding the imaging operator is to study the 
canonical left and right projections from the canonical relation C C T*Y x T*X 
to T*Y and T* X, respectively. 
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LOY TX (59) 
In order to motivate the results that follow, let us consider the following example. 


Example 12. Let us consider a simple example from SAR imaging. This example 
will help us explain via microlocal analysis, some of the artifacts introduced by 
image reconstruction operator in 6. 

Assume that a colocated transmitter/receiver traverses the straight trajectory 
y(s) = (s,0,/) with h fixed. The forward operator in this case is 


iw 2 /(x-s)2+y2 +h? 
Prot= fe ( a ar ) Abs ssa Ge phdndiden: 


Using (51) and (52), the canonical relation of this operator is easily computed to be 


2 2 _ 
C= s,—VJV(x —sP + y2 + h?,-@ = ds + dt ]; 
Co co J/(x— s+ y2 +7? 


2 _ 
x,Y,— a <= dx + = dy ; 
co \ J—s)? + y2 +h? V(x —s)? + y2 +h? 


Now using Hormander-Sato Lemma, one sees that 


WE(P*P) C 


x,y, @ X—S ie 2 y iy 
co\ V(x—s? + y2 +h? Vax-srt+y2 +h? 


2w z-Ss Ww 
Z,W, dz+ dw]: 
co \ J/@-sP tw +h? Vg-s? +w2 +h? 


Vix—s?+ y2+h? = J(c—s)? + w? + h? and 


x—s zZ—s 40 
= 0) ; 
V@-sP+y4+h J@-sP+wt+h? 


This implies that (z, w) = (x, y) or (z, w) = (x, —y). The first equality contributes 
to the diagonal relation of the wavefront set of P*P, while the second contributes 
to the relation formed by reflection about the x-axis. In other words, 


WE(P*P) C AUG, where A = {(x, y, dx + ndy : x, y,Edx + ndy)} 
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and G = {(x, y,&dx + ndy; x,—y, &dx — ndy)}. 


The presence of the set G as in the above example (the non-diagonal part) 
indicates that the imaging operator introduces artifacts in the reconstructed image. 
A detailed study of the class of distributions as in the example above, called 1?! 
classes, was introduced in [31, 32,39, 64]. These classes of distributions have come 
up in the study of several imaging problems including X-ray, seismic, SAR, and 
electron tomography [1, 13, 21-24, 30-32, 54, 69]. In instances where the imaging 
operator introduces artifacts, it is of interest to determine whether the artifacts are of 
the same strength in a suitable sense as that of the true singularities and whether the 
artifacts can be suppressed or displaced from the true singularities. These questions 
are answered in the references given above. 


Monostatic SAR Imaging 

In monostatic SAR imaging, the transmitter and receiver are located at the same 
point. In other words, yr(s) = yr(s). Nolan and Cheney in [70] investigated 
the microlocal properties of the forward operator ? and the associated image 
reconstruction operator P*P. Using microlocal tools, synthetic-aperture inversion 
in the presence of noise and clutter was done in [102]. Other imaging methods 
in the context of SAR imaging, again using microlocal tools, were considered in 
(99, 101]. The forward operator P was further investigated by Felea in [19], and 
she made a detailed analysis of the image reconstruction operator for various flight 
trajectories. Felea in [19] showed that for y(s) = (s,0,) with h > 0 fixed, the 
operator P*P belongs to 1~”°(A, G) where A = {(x,&,x,&) € T*X x T*X \O} 
and G is the graph of the function y(x1, x2, &, &) = (%1, —X2, &, -&). If y(s) = 
(cos s, sin s,/), it was shown in [20] that P*P € L(A, G), where G is a 2- 
sided fold. Mappings with singularities (such as folds and blowdowns) are defined 
in [29, 37]. Furthermore, in [20], Felea showed that in some instances such as the 
flight trajectory being circular, the artifact singularities of the same strength as the 
true singularities can be displaced far away from the true singularities, and those that 
are not displaced are of lesser strength compared to the true singularities. In [91], 
the authors show that cancelation of singularities, that is, only certain singularities 
are recoverable, can occur even in curved flight paths. 


Bistatic SAR Imaging 

Some recent results by the authors and their collaborators investigating the microlo- 
cal properties of transforms that appear in bistatic SAR imaging are now described. 
For related work, the reader is referred to [100]. 


Common Offset Bistatic SAR Imaging 
In common offset SAR imaging, the transmitter and receiver travel in a straight line 
offset by a constant distance at all times. More precisely, let 
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yr(s) = (s + a@,0,h) and yr(s) = (s — a, 0,h) 


be the trajectories of the transmitter and receiver, respectively, with a and h fixed 
positive quantities. A detailed microlocal analysis of associated forward operator P 
and the imaging operator P*P was done in [54]. The authors obtained the following 
results analogous to the ones obtained by Nolan and Cheney in [70] and Felea in [19] 
for the monostatic case. 


Theorem 19 ([54]). Let yr(s) = (s + a,0,h) and yr(s) = (s — a,0,h) where 
a > 0,h > 0 are fixed. The operator P defined in (57) is an FIO. The canonical 
relation C associated to P defined in (58) satisfies the following: the projections m1, 
and Ip defined in (59) are a fold and blowdown, respectively. 


Theorem 20 ((54]). Let P be defined with yr and yr given in Theorem 19. Then 
P*P € I>°(A,G), where A is the diagonal relation and G is the graph of the map 


X(%1, x2, &1, 2) = (41, —X2, €1, —&2). 


One important consequence of this result is that the artifacts given by the graph G of 
the map 7 have the same strength in a Sobolev sense as that of the true singularities 
given by the diagonal relation A. 


Common Midpoint SAR Imaging 

In common midpoint SAR imaging, the transmitter and receiver travel in a straight 
line at a constant height above the ground at equal speeds away from a common 
midpoint. The trajectories of the transmitter and receiver for the common midpoint 
geometry considered here are 


yr(s) = (s,0,h) and yr(s) = (—s,0,/). (60) 


A detailed microlocal analysis of the forward operator (58) associated to yr and 
Yr and the imaging operator P*P was done in [1]. In contrast to the results in 
[19,54, 70], here the canonical relation C associated to P is a 4-1 relation, and this 
is reflected in the fact the canonical left and right projections zz, and zr drop rank 
on a union of two disjoint sets. More precisely, one obtains the following results for 
the forward operator and the imaging operator, respectively. 


Theorem 21 ({1]). Let P be as in (57) with the trajectories given by (60). Then 
P is an FIO and the canonical relation associated to P defined in (58) has global 
parametrization 


(0, 00) x (R” \ 0) x (R \ 0) 3 (5,.x1,2,@)  C, 


and it satisfies the following: the left and right projections m1; and 1p drop rank 
simply by one on a set 3 = & U Sy where in the coordinates (s,x,w), © = 
{(s,x1,0,@) : s > 0,|x1| > &,@ 4 0} and Ly = {(s5,0,x2,@) : 5 > 0,|x2| > 
e',w 4 O} for 0 < €' small enough. The canonical relation C associated to P 
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satisfies the following: the projections m;, and mp defined in (59) are a fold and 
blowdown, respectively, along >’. 


Theorem 22 ({1]). Let P be as in (57) with the trajectories given by (60). Then 
P*P can be decomposed into a sum belonging to 17"°(A, Gy) + 17" °(A, G2) + 
1P°"°(G,, G3) + 17"°(Go, G3), where G; fori = 1,2,3 are the graphs of the 
following functions y; fori = 1,2,3 on T*X: 


WA%, €) = (%1, —X2, 1, -€2), X20, &) = (— 41, X2, —€1, &2) and x3 = ¥1 © ZX. 


As with the common offset case, the artifacts given by graphs of the maps 71, 
X2, and y3 have the same strength in a Sobolev sense as that of the true singularities 
given by the diagonal relation A. 


6 Conclusion 


Finally, some important themes of this chapter will be highlighted. 

Microlocal analysis can help understand strengths and limitations of tomographic 
reconstruction methods. For limited data X-ray tomography, microlocal analysis 
was used in this chapter to show which singularities of functions will be visible 
depending on the data, and reconstructions in this chapter illustrate the paradigm. 

Each of the reconstruction methods described in this chapter is of the form 


L=P*DP 


where the forward operator (operator modeling the tomography problem) P is a 
Fourier integral operator and P* is an adjoint and D is a pseudodifferential operator. 
In SAR imaging, D = Id is the identity operator and the reconstruction method is 
P*P — the normal operator. Since the operator D is a differential or pseudodifferen- 
tial operator, it does not add to the wavefront set of Pf, WF(DP f) C WF(P f). 
If D is elliptic (on the range of ?, which is true in the cases considered here), 
WF(DP f) = WF(P f). Then, one needs to understand what P* does, and this is 
determined by the structure of C'oC by the Hérmander Sato Lemma (Theorem 16): 


WE (P* (DP f)) Cc (C' oC) o WE(f). 


However, in limited angle tomography, because R7 j;,, is not a standard FIO, 
Ri jimP Rx adds singularities along lines at the ends of the limited angular range. 

In the case of SAR and conical tilt ET, C’ oC is more complicated; it includes A 
and another set. In conical tilt ET, this second set generates A(/), the extra artifacts 
given in Theorem 18. Also, for conical tilt ET, that theorem implies that a well- 
chosen differential operator ((—Ds) rather than (—As)) decreases the strength of 
the added singularities; they are visible if one looks carefully at the reconstructions, 
but they are smoother than when (—As) is used. 
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The only exact reconstruction method presented in this chapter is FBP (Theo- 
rem 9). The other algorithms, such as Lambda CT, involve differential operators and 
backprojection. Microlocal analysis was used to demonstrate that they do recover 
many (or all) singularities of the object. 

Recovering singularities does not recover the object. So, these algorithms are not 
useful when one needs density values, such as in distinguishing tumor cells from 
benign cells in diagnostic radiology. However, in many cases (e.g., ET or industrial 
nondestructive evaluation) one is interested in the shapes of regions, not actual 
density values, so knowing the location of singularities is useful. The algorithm must 
be designed so that it clearly shows singularities in the reconstruction. For example, 
an algorithm that turns jump discontinuities in the object into discontinuities of 
a derivative in the reconstruction might not provide a clear picture of the object. 
Lambda CT and the algorithms given here for conical tilt ET actually accentuate 
singularities; they make the singularities more apparent since they are operators of 
order one (like a first derivative). The implementation smooths out the derivative 
since it uses numerical differentiation. 

In conical tilt ET and SAR the reconstruction methods, that included a backpro- 
jection (adjoint, P*), produce added artifacts (as shown in Figs. 5 and 6), and these 
were explained using microlocal analysis. 

Microlocal analysis will not make a bad algorithm good, but it can show 
that some limitations in reconstruction quality are intrinsic to the underlying 
tomographic problem. It can point to where reconstruction methods need to be 
regularized more strongly because of intrinsic instability in the specific tomography 
problem. In summary, microlocal analysis can be used to understand practical and 
theoretical issues in tomography. 
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Abstract 


In this chapter, we present the mathematical formulation of the inverse Radon 
transform and of the inverse attenuated Radon transform (IART), which are used 
in PET and SPECT image reconstruction, respectively. Using a new method for 
deriving transform pairs in one and two dimensions, we derive the inverse Radon 
transform and the JART. Furthermore, we discuss an alternative approach for 
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computing the Hilbert transform using cubic splines. This new approach, which 
is referred to as spline reconstruction technique, is formulated in the physical 
space, in contrast to the well-known filtered backprojection (FBP) algorithm 
which is formulated in the Fourier space. Finally, we present the results of 
several rigorous studies comparing FBP with SRT for PET. These studies, which 
use both simulated and real data and which employ a variety of image quality 
measures including contrast and bias, indicate that SRT has certain advantages in 
comparison with FBP. 


1 Introduction 


Positron emission tomography (PET) is an important, noninvasive, nuclear medicine 
modality that measures the in vivo distribution of imaging agents labeled with 
positron-emitting radionuclides. The importance of PET in detecting, staging, and 
monitoring the progress of several diseases has been established in a plethora 
of rigorous clinical studies [1]. Furthermore, small-animal PET is becoming an 
essential imaging modality for preclinical research [2-4], as well as for drug 
development and discovery [5]. 

Single-photon emission computed tomography (SPECT) is the most widely 
available diagnostic imaging technique which uses short-lived radiographic iso- 
topes. In particular, it is extensively used in cardiology. For example, in 2012, nine 
million myocardial perfusion scintigraphic imaging studies were performed in the 
USA. 

Image reconstruction is an essential component in tomographic medical imaging, 
allowing a tomographic image to be obtained from a set of two-dimensional 
projections. The existing image reconstruction methods can be classified into two 
broad categories: (a) analytic methods and (b) iterative (or algebraic) methods. 
In what follows, we will discuss only analytic methods. In this respect, we first 
consider a simple model for SPECT: let jz(x1, x2) denote the gamma-ray attenuation 
coefficient of the tissue at the point (x1, x2). This means that a gamma ray traveling 
a small distance At at (x, x2) suffers a relative intensity loss 


Al 


We denote by J, the initial intensity of a gamma ray and by J; the measured 
intensity after undergoing attenuation through tissue of length L; the above equation 
can be written in the form 


Ty = I, exp =] L(t) ark . (2) 
L 


This equation expresses the well-known Beer’s law. 
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Let f(x1, x2) denote the distribution in the tissue of the radioactive material 
under consideration, and let L(x) denote the part of the ray from the tissue to the 
detector. Then, in SPECT, it is assumed that the following integral, /spgcr, is known 
from the measurements: 


Tspecr = / exp — f past f dt, (3) 
L L(x) 


where the integral is measured over a finite number of lines L. Equation (3) is valid 
under the following assumptions: 


(a) The lines L are lying within a specified imaging plane, 

(b) The imaging system has perfect imaging characteristics, 

(c) The detector is collimated and can pick up radiation only along the straight lines 
L, 

(d) The collimator has an infinitely high spatial resolution, 

(e) The attenuation coefficient jz obeys Eq. (2), 

(f) There is no scatter radiation in the object. 


In PET, the integral in Eq. (3) simplifies to 


Tper = exp tit} ff dt, (4) 


where jz denotes the Radon transform of [A, i.e., 


p= fo ude (5) 
L 


In order to derive Eq. (4) from Eq. (3), we recall that positrons eject particles 
pairwise in opposite directions, and in PET the radiation in opposite directions is 
measured simultaneously. Thus, 


/ pas = | pas+ | pads= | was (6) 
L(x) L—(x) L+(x) L 


and Eq. (3) becomes Eq. (4). 

In both PET and SPECT, a projection is formed by combining a set of line 
integrals along a line L. Here, we will consider parallel-ray integrals at a specific 
angle 0, i.e., we will assume a parallel-beam geometry. 

We recall that the Radon transform of the X-ray attenuation coefficient denoted 
by fcr is measured via computed tomography (CT). Using the values of jucr, it is 
possible to estimate f(x, x2) by appropriate scaling (this is necessary due to the 
energy difference between X-rays and gamma rays). 
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Equation (4) implies that in PET one needs to reconstruct a function f from the 
knowledge of its Radon transform denoted by f/f, 


f=f rae (7) 


where 7 can be computed via 


f = Iver exp {fi}. (8) 


Hence, both CT and PET involve the computation of a function from the knowledge 
of its Radon transform. More specifically, in CT one needs to compute /icr from the 
knowledge of jicr, whereas in PET one needs to compute f from the knowledge of 
Ipgr exp {(1}. The relevant formula, known as the inverse Radon transform, is given 
by 


1 20 ; 
— (0x, =18,) | e!9 T(x1, x2, 0) dé, (9) 
0 


SF (1, x2) = de 


where J is defined in terms of f by 


I F(p, 0) d 
I¢s1.42.0) = =f FEO ose <2", 10) 
ix J, p— (x2 cos 6 — x; sin) 


and throughout this chapter, 6 denotes the principal value integral. 

Let ve denote the right-hand side (RHS) of Eq. (3). We call i the attenuated 
Radon transform of f (with attenuation specified by the given function jz). Hence, 
SPECT involves the computation of a function f from the knowledge of its 
attenuated Radon transform hi and of the function jz. The relevant formula is known 
as the inverse attenuated Radon transform (IART). 

According to Eq. (9), the numerical implementation of the inverse Radon trans- 
form involves the computation of the Hilbert transform (Hf )(p) of a given function 
F (pe), where 


fe) 


1 
(HAY) = —@ - a) 


The most well-known method for this computation uses the fast Fourier transform 
technique, exploiting the fact that the Hilbert transform is a convolution. The 
application of this method to the computation of the inverse Radon transform is 
called filtered backprojection (FBP). 

Most PET systems have options for both FBP and Ordered Subset Expectation 
Maximization (OSEM), which is the predominant iterative technique. 
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The analytical approach to SPECT requires the numerical implementation of 
the IART. However, since no analytical formula was available until recently for 
the IART, currently the FBP implementation to SPECT generally makes the crude 
approximation of jz = 0 or incorporates a uniform attenuation map [6]. On the other 
hand, since OSEM is based on an iterative statistical approach, it does not require 
the analytical inversion formula, and hence OSEM considers the actual values of ju. 
Hence, in SPECT, OSEM produces more accurate images than FBP. However, for 
several reasons including speed (1s vs. 2.5 min for a typical study), FBP is still used 
extensively. Indeed, most SPECT systems have options for both FBP and OSEM, 
and both are used clinically. Actually, some clinicians prefer FBP for quantification. 

In this review, we will discuss an alternative approach for computing the Hilbert 
transform. This approach, in contrast to FBP which is formulated in the Fourier 
space, is formulated in the physical space, and it is based on “custom-made” cubic 
splines. We will refer to the application of this approach to the computation of the 
inverse Radon transform and to the IART as spline reconstruction technique (SRT) 
for PET and SRT for SPECT, respectively. 

The results of rigorous studies comparing FBP with SRT for PET are published in 
[7]. In these studies, which use both simulated and real data, by employing a variety 
of image quality measures, including contrast and bias, it is established that SRT 
has certain advantages in comparison with FBP. As a result of these studies, SRT is 
now included in STIR (Software for Tomographic Image Reconstruction) [8], which 
is a widely used open-source software library in tomographic imaging (FBP and 
SRT are the only algorithms based on analytical formulae which are incorporated in 
STIR). 

Rigorous studies comparing SRT for SPECT with FBP and OSEM are work in 
progress. Preliminary results suggest that SRT is preferable to FBP and also that 
SRT is comparable with OSEM; see Fig. | [9]. 

The clinical importance of PET and SPECT, as well as the mathematical 
formulation of the inverse Radon transform and of the IART, is further discussed 
in the following section. A new method for deriving transforms in one and two 
dimensions was introduced in [10]. Using this method, both the inverse Radon 
transform and the IART are derived in Sect.3. The SRT for PET is presented in 
Sect. 4, and the studies of Kastis et al. [7] comparing SRT with FBP for PET are 
reviewed in Sect. 5. SRT for SPECT is briefly discussed in Sect. 6. 


2 Background 
The Importance of PET and SPECT 
In 1964, the research group of Dr. David E. Kuhl, known as the “Father of Emission 


Tomography,” developed the Mark II SPECT series, which is a single-emission 
computed tomography camera. Using this unit, this group succeeded in producing 
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SRT Fused with CT System OSEM Fused with CT 


Fig. 1 Reconstructions of two consecutive slices (a and b) of the myocardium acquired with 
SPECT fused with CT scan. The images on the right are obtained by the currently used commercial 
software and on the /eft with SRT 


the world’s first tomographic images of the human body. This achievement was 
considerably earlier than Godfrey Hounsfield’s development of the X-ray CT 
device in 1972. A crucial element in the evolution of emission tomography was 
the choice of the radioisotope injected in the patient. An important such class is 
those radioisotopes that emit positrons. An emitted positron almost immediately 
combines with a nearby electron; they annihilate each other, emitting in the process 
two gamma rays traveling in nearly opposite directions. PET cameras detect these 
gamma rays. 

FDG ('8F-2-deoxy-2-fluoro-D-glucose) is the most suitable positron emitter that 
can be used with humans. It is a deoxyglucose analog with the normal hydroxyl 
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group in the glucose molecule substituted with the radioactive isotope '*F. FDG 
is taken up by high-glucose-using cells, and it is metabolically trapped following 
phosphorylation by the hexokinase enzyme [11]. Thus, the measurement of the 
positron-emitting '8F provides a quantitative measure of the glucose consumed 
in the corresponding tissue. Glucose metabolism in cancerous tissue is higher 
than in normal tissue. Thus, PET-CT devices are proving to be highly effective 
in the early detection of cancer. For example, among 91 patients with non-small- 
cell lung cancer, 38 pathologically confirmed mediastinal lymph node metastases 
were missed by CT, whereas among 98 similar patients, only 21 metastases were 
missed by PET-CT [12] (this means that in the former case, there were 38 futile 
thoracotomies, whereas in the latter case, there were only 21). Furthermore, using 
more specific positron emitters, it is possible to obtain additional useful information. 
For example, using F-fluoro-17-f-estradiol, it is possible to access in vivo the 
density of estrogen receptors as well as to monitor the response of the treatment 
for estrogen receptor-positive breast cancer. Similarly, using '8F-annexin V, it is 
possible to follow tumor cell apoptosis (death) in vivo, as well as to monitor 
treatment response to various cancer types. It should also be noted that PET- 
CT is useful for prognosis. For example, the 2-year failure-free survival rate of 
patients with positive scans after chemotherapy treatment for early-stage Hodgkin’s 
lymphoma was 69 %, as compared with 95 % of patients with negative scans [13]. 

In addition to oncology, PET is now used in several other areas of medicine, 
including cardiology and neurology. As an example of the usefulness of PET in 
neurology, we note that PET with the use of the Pittsburgh compound B(PIB) can 
be used to quantify the concentration of amyloid-beta (Af) deposition in the brain, 
a precursor of Alzheimer’s disease [14]. 

SPECT is used extensively in cardiology, oncology, and neurology. In cardiology, 
SPECT is part of the common stress-testing procedures for the evaluation of chest 
pain [15]. As an example of the use of SPECT in oncology, we mention the study 
in [16], where lymphoscintigraphy was performed using SPECT-CT to establish 
a preoperative road map of lymph nodes that are at risk for metastatic melanoma 
and to facilitate intraoperative identification of the “sentinel” nodes (the rationale 
for sentinel-node biopsy relies on the concept that different regions of the skin 
have specific patterns of lymphatic drainage to the regional nodes, and for a given 
region of the skin, there exists at least a specific node, the sentinel node, in 
which lymphatic vessels drain first, and it is this node the most likely first site of 
metastasis). 

In neurology, SPECT can be used to image the distribution of the blood flow in 
the brain, and this has been used in a plethora of pathological situations. Further- 
more, by employing specific radioisotopes, it is possible to obtain useful information 
about several diseases. For example, using SPECT with !77]-labeled CIT, it is shown 
in [17] that although levodopa treatment is highly effective as dopamine replacement 
in Parkinson’s disease, this treatment apparently downregulates the endogenous 
dopamine transporters. 
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Fig. 2 Parallel-beam projections through an object with attenuation coefficient z(x1, x2). The 
local and Cartesian coordinates of the mathematical formulation are indicated 


The Mathematical Foundation of the [ART 


Consider a line L specified by two real numbers p and 6, where —oo < p < co and 
0 < 6 < 2m (Fig. 2). A unit vector along this line is given by 


e! = (cos 6, sin@). (12) 
A unit vector perpendicular to L is given by 
sls i 
e~ = (—sin6,cos@). (13) 


A point x on this line can be expressed in the form 


x=pet+rel. (14) 

Hence, 
xX; = tTcos@ — psin#, (15) 
x2 = tsin@ + pcos6. (16) 


Solving these equations for (t, e) in terms of (x1, x2, 8), we find 


x2 sind + x; cos@, (17) 


II 


p = x2 cos 8 — x; sin @. (18) 
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Writing (x1, x2) in terms of the local coordinates (p, @), it follows that the equation 
defining the Radon transform, f, of a function f takes the form 


f(0.8) = f f («cos 6 — psind,t sin @ + pcos 6) dt, 
—0 <p<co, 0<6@<2xz. (19) 


Similarly, the following equation provides the definition of the attenuated Radon 
transform, f,, (with attenuation specified by jz), of a function /: 


co 


fulp.8) = | exp |- | iL(s cos 8 — psin 6, ssin@ + pcos @)ds} x 


—Co 


f(tcos@ — psiné,tsin6d + pcosé)dt, -co<p<oo, 0<86@ <27.(20) 


The inverse Radon transform, expressing f in terms of f , is given by Eqs. (9) 
and (10). Similarly, the IART, expressing f in terms of f,, and 4, is given by 


1 20 ; 
F012) = Fs, = 19s) | e! T(x1, x2, 0)d0, —00 < X1,X2 < 00, 
0 


(21) 
where the function J is defined by 


J (x1, 2,0) = exp} f LL(s cos 6 — psin 6, s sin + pcos @)ds 


T 


x Lu (p, 0) fulp, 8) T=X7 sind+x1 cosé (22) 
p=x2 cos O—x, sind 
and the operator L,,(p, @) is defined by 
Ly(p, 0) = exp{P™ fi(p,@)} P” exp{—P fu(p, ®)} 
+ exp {—P* ji(p, 0)} P* exp{P*fi(p, 4}, (23) 


with /2(p,@) denoting the Radon transform of 4(x;,.x2) and the operators P* 
denoting the usual projection operators in the variable p, i.e., 


(P*g)(p) = +2) 4 1 f_ g(r) ae 
=o f =p 


—-0 <p<o. 24 
o." Bin = (24) 


A General Methodology for Constructing Transform Pairs 


The study of nonlinear integrable equations led to the emergence of a general 
method for deriving transform pair in one and two dimensions [10]. In particular, 
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using this method, it was shown in [10] that the two-dimensional Fourier transform 
of a function u(x, x2) can be constructed via the spectral analysis of the equation 


Ww iow 
+ ck — kW = u(x, x2), (%1,%2) € R*, k EC, (25) 
Ox, 0x2 


where W(x,, X2,k, k) is a scalar function and bar denotes complex conjugate. 
R. Novikov and one of the authors re-derived in [18] the Radon transform by 
performing the spectral analysis of the equation 


1 1\ ow 1 1\ ow 
k k- = R,kec. (2 
5 ( + z) ve +5 ( :) Ox F(%1,%2), (%1,%2) ER°, K EC (26) 


Although the Radon transform can be derived in a much simpler way by using 
the two-dimensional Fourier transform, the advantage of the derivation of [18] was 
demonstrated later by Novikov [19] who showed that the inverse attenuated Radon 
transform can be derived by applying a similar analysis to the following slight 
generalization of Eq. (26): 


Lane Oe 
5 iE) ba OF [a 


= f(x1, x2), (41,22) ER’, kK EC. (27) 


3 The Inverse Radon Transform and the Inverse Attenuated 
Radon Transform 


In what follows, we discuss an algorithmic approach for the construction of a 
transform pair { f, 7 }. The relevant analysis, which is usually referred to as spectral 
analysis, involves two main steps: (i) solve a given eigenvalue equation in terms 
of f. If k denotes the eigenvalue parameter, this involves constructing a solution 
W of the given eigenvalue equation which is bounded for all complex values of k. 
This problem will be referred to as the direct problem. (ii) Using the fact that W 
is bounded for all complex k, construct an alternative representation of W which 
(instead of depending on f) depends on some “spectral function” of f denoted by 
7 . This problem will be referred to as the inverse problem. 

It turns out that the inverse problem gives rise to certain problems in complex 
analysis known as the Riemann-Hilbert and the d-bar problems. Indeed, for certain 
eigenvalue problems, the function W is sectionally analytic in k, i.e., it has different 
representations in different domains of the complex k-plane, and each of these 
representations is analytic. In this case, if the “jumps” of these representations 
across the different domains can be expressed in terms of i , then it is possible 
to reconstruct W as the solution of a Riemann-Hilbert problem which is uniquely 
defined in terms of Ff . However, for a large class of eigenvalue problems, there 
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exists a domain in the complex k-plane where W is not analytic. In this case, if 
ow/ dk can be expressed in terms of 7 , then W can be reconstructed through the 
solution of a d-bar problem which is uniquely defined in terms of f. . 

We recall that the classical derivation of transform pairs involves the integration 
in the complex k-plane of an appropriate Green’s function. However, this derivation 
is based on the assumption that the Green’s function is an analytic function of k 
and it also assumes completeness. The assumption of analyticity corresponds to 
the case that W is sectionally analytic. Therefore, the approach reviewed here has 
the advantage that not only it provides a simpler approach to deriving classical 
transforms avoiding the problem of completeness, but also it can be applied to 
problems that the associated Green’s function is not an analytic function of k. 

In addition to the construction of the Inverse attenuated Radon transform, the 
approach reviewed here has led to the construction of the X-ray fluorescence 
computed tomography [20]. 


The Construction of the Inverse Radon Transform 


Let S(R’) denote the space of Schwartz functions. Define the Radon transform 
fp. 6) of the function f(x;,x2) € S(R*) by Eq. (19). Then, for all (x1, x2) € R’, 
F(X, X2) is given by Eq. (9). 

We will derive the Radon transform pair by performing the spectral analysis 
of the eigenvalue Eq. (26). In order to solve the direct problem, we first simplify 
Eq. (26) by introducing a change of variables from (x,,x2) to (z,Z), where z is 


defined by 
1 1 1 1 
= k- —i=(k : 28 
z = ( 7). 5 ( +7) (28) 


and z follows via complex conjugation, i.e., 


1 - 1 1 (- 1 
a (F-z)-3 (E+ zs (x1,22) ER’, kEC. 


Using the identities 


1 1 1/- 1 
= k- --—|k--= Ee 2 
xy 2i ( 7) a 21 ( 7): 2?) 
1 1 1/- 1 
a k 2 k = Z 
Ox, = ( +e)a 5 ( +7) o (30) 


and 


Eq. (26) becomes 
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aw 
(Ik) eC, 4254) = f(x1,%2), (%1,%2) ER’, k EC, (31) 


where the function v(|k|) is defined by 


1 
k))=— kl 3 32 
ved = = (Geer) G2) 
We supplement Eq. (31) with the boundary condition 
1 
v=0(+). Z7>O&. (33) 
z 


In order to solve Eqs. (31) and (33), we will use the Pompeiu formula, namely, 


il IEE aes 
se= eee ' (t,8) 2eD, (34) 


2in aD C-z 


where dD denotes the boundary of the domain D. Using this formula, we find 


1 d 
| TOD EENER. oe che: Bee |kK| Al. (35) 
R2 


~ Oni v(kl) 2—z 


Hence, using the identity 


1 
dzA\dz= a ler- 5 dx,dxy (36) 


IkP 


it follows that for all (x,, x) € R? andk € C, |k| 4 1, W satisfies 


if - id. 
Horak) = sosen( Te -ke) ff foxy. en 


If &k is either inside or outside the unit circle, the only dependence of W on k is 
through z’ and z; thus, W is a sectionally analytic function with a “jump” across the 
unit circle of the complex k-plane. Equation (37) provides the solution of the direct 
problem. 

In order to solve the inverse problem, we will formulate a Riemann-Hilbert 
problem in the complex k-plane. In this respect, we note that Eq. (37) implies 


v=0(;). k>o. (38) 


Furthermore, we will show that for all (x;,x2) € R’, W satisfies the following 
“Jump” condition: 
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wt—w =i(Hf)(p,0), -wo<p<o, 0<6<2z, (39) 


where H denotes the Hilbert transform in the variance p defined in Eq. (11). This 
equation is a direct consequence of the following equations: let ™t and W~ denote 
the limits of W as k approaches the unit circle from inside and outside, i.e., 


Ww = lim ¥ (x1, x2, (IF ee"’). (40) 


Then, for all (x), x2) € R’, 


ve = £(P* fyip.6)— f F(p,s,0)ds, -co<p<o, O<8@<2z, 


where P~ denote the usual projectors in the variable p, i.e., a 
(P* fyi) = +7 + Sano) (42) 
and F denotes f in the coordinates (p, T, 0), i.e., 
F(p,t, 0) = f(tcos 6 — psin@, t sin@ + pcos@). (43) 
Indeed, in order to derive Eq. (41), we note that the definition of z implies 
(c-z)= = («-z) (ax) 5 (K+ z) (x2 — x4). (44) 
Let 
kt =(1—eje”®, k~> =(1+ e)e’®, 0< 0 < 2m, &>0. (45) 
Thus 
(i a =) = (1—ee? FA +e)e? + O€?) (46) 
and similarly for (k~ # 1/k7). 
Hence, for computing Y~ via Eq. (37), we must use the formulae 
Zz —z= (x; — x1) sin 6 — (x4 — x2) cos 
tie [(x, — x1) cos 6 + (x5 — x2) sin6] + O(e7). (47) 


We recall that solving Eqs. (15) and (16) for (p,t) in terms of (x1, x2) we find 
Eqs. (17) and (18). The Jacobian of this transformation equals 1, hence dx;dx2 = 
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dpdt. Replacing in Eq. (37) z— Zz by the RHS of equation (47) and then changing 
variables in the resulting equation from (x/,x) to (’,t’) and from (x), x2) to 
(p, T), we find 


1 F / / d / / 
y+ == tim ff (ee AOE (48) 
2inx «>0J Jr p’—[o + ie(t’ —7)] 


In order to evaluate this limit, we must control the sign of t’ — t. This suggests 
splitting the integral over dt’ as shown below: 


Game d gl [ Fdt! ae Fdt! ae 
“= —-— lim ; 
Dit 0) oo lJ-ooP’—lp Lie’ —d) J, pp —lpLie’—D]\” 
(49) 


In the first and second integral above, t’ — t is negative and positive, respectively: 
hence 


1 Tv 
yt = | {FiF (p, 0’, 0) + (HF)(p, 0’, 0)} dt’ 
T J—oo 
1 co 
ar / {+iF (p, t',0) + (HF)(p, 1’, 0)} dr’. (50) 
Adding and subtracting in the RHS of this equation the expression 
1 [o,e) 
+ = / (F)iF (p,t', A)dt’, (51) 
1 t 


we find Eqs. (41). 

The sectionally analytic function W satisfies the estimate (38) and has a jump 
across the unit circle; thus, for all (x;,x2) € R?, it admits the following 
representation: 


,kEC, |k| #1, -w~ <p<oo. 


(52) 
Replacing in this equation Yt — W~ by the RHS of (39), we find the following 
expression valid for all (x), x2) € R?: 


1 Wt We) (0, oie! do" 
Qin i ef _k 


y= 


2x i0! a ’ ’ 
a ; INO EY ¢ oes tee to epee. 63) 
2ix Jo eo _k 
This expression provides the solution of the inverse problem. 
Using Eqs. (26) and (53), it is straightforward to express f in terms of f . One 
way of achieving this is to replace in Eq. (26) W by the RHS of Eq. (53). A simpler 
alternative way is to compute the large k behavior of WY: Equation (53) implies 
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1 2 re _ 1 il 
v~ laf e (H f)(p.0)d0} 7+0(z). k= So: (54) 


Substituting this expression in Eq. (26), we find that the O(1) term of Eq. (26) yields 


Qn 
jo (2. + rss) / e!9(H f\(0,0)d8, (55) 
4in I 0 


which is Eq. (9). 


The Construction of Inverse Attenuated Radon Transform 


Let k* denote the limiting values of k € C as it approaches the unit circle in 
the complex k-plane from inside and outside the unit circle; see Eq. (45). Let z be 
defined in terms of (x;, x2) € R? and k € C by Eq. (28), and let v(|k|) be defined 
by Eq. (32). Then 


slim, $=" (a) eR Fit, a) | F(p,s, 0)ds, 


-o<p<o, 0<6<2n, (56) 


where f is the Radon transform of f (see Eq.(19)), P~ are the usual projectors 
in the variable p (see Eq. (42)), (p, tT) are defined in terms of (x), x2) by Eqs. (17) 
and (18), and F denotes f in the variables (p, t, @) (see Eq. (43)). Indeed, Eq. (56) 
is direct consequence of Eqs. (31) and (41). 

It turns out that the derivation of the attenuated Radon transform pair is a direct 
consequence of Eqs. (27) and (56). Indeed, define the attenuated Radon transform 
i (p, @) of the function f(x1, x2) € S(R’) by Eq. (20). Then, f(x1, x2) is given by 
Eq. (21). 

In order to derive the above result, we note that Eq. (27) can be rewritten in the 
form 


rN nay ee (57) 
dz v v 
Hence 
d a4] Fay 
alee | = ), (58) 
or 


wes () = gz! (4) ci) , (%1,%2) ER’, keEC. (59) 
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This equation provides the solution of the direct problem, i.e., it defines a sectionally 
analytic function W with the estimate (38), which has a jump across the unit circle of 
the complex k-plane. Hence, W is given by Eq. (52). All that remains is to determine 
the jump W+ — W~. This involves computing the limits as k — k* of Oz '(u/v); 
thus, it can be achieved using Eq.(56). Let M denote y in the variables (p, T, 6): 
then using in Eq. (59) the identity (56) with f replaced by j2, we find 


we P fie [ M(p,s,0)ds — lim, az! cer? he-S M(p,s,0)ds . (60) 
aS an 


For the computation of the RHS of this equation, we use again Eq. (56) where f 
is now replaced by /f times the two exponentials appearing in the curly bracket of 
Eq. (60). Hence, the RHS of Eq. (60) yields 


= lo.) an en 
— Ptet? ‘ff F(p,t', 0)et? fe Sr M(p.s.8)ds qo! (61) 


The term exp[+P* fi] is independent of t’; hence, this term comes out of the 
integral f pass and furthermore, the same terms appear in the left-hand side of 
Eq. (60). Hence, 


wtew- = -J, (62) 


where J is defined in Eq. (22). Then Eq. (52) yields 


1 2% e@ T(p,t, 0’)dd’ 
v=-— | elf _k , (65) 
Hence, 
1 2n 
y= = | e'" I(p,t, 0)d0 it O k > ov. (64) 
20 0 = 


Substituting this expression in Eq. (27), we find that the O(1) term of Eq. (27) 
yields (21). 


4 SRT for PET 


According to Eq. (9), the inverse Radon transform can be written in the form 


dé, (65) 


p=x2 cos O—x1 sind 


fim) = ie [ete 6) 
fale Qin \ dx, aes 0 ne 
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where F (p,@) denotes half the Hilbert transform of 7 (p, 0) with respect to p, 
namely, 


1 £& f(r,6) 
F(p,) = =— — 


dr, —00 <p <0, 0<6 <2n. (66) 


Inserting the operator oon -i 5) inside the integral in the right- hand side of 
Eq. (65), we find 


(5 -ix-) F(x2 cos 0 — x; sin 8, 9) 
Ox 0x2 


OF (p, 0) 


= —(sin@ +i cos@) (67) 


p=x2 cos O—x, sind 


Using Eq. (67) in Eq. (65), we find 


1 7" [aF(p,0 
Ff (x1, X2) = - i (p J d0, —O < X1,X%2. < OO. 
2n 0 dp p=x72 cos O—x, sinO 
(68) 


For the numerical calculation of F(p, 6), we assume that f (pe, 8) has support in the 
interval —1 < p < | and that f(p, @) is given for every 6 at the n points {p;}/. We 
denote the value of a at p; by 7 i.e., 
f=f(o.%, eo €Ell, O0<0<2n, i=1,...,n. (69) 
Furthermore, we assume that 
f(-1,6) = f,6) = 0. (70) 


In the interval p; < p < pj+1, We approximate 7 (p, 8) by cubic splines: 


f (0,9) = ai (0) + bi(0)p + ci (0) 0? +d: (9)—°, pi<p<pi41, (71) 


0<0<2n, i=1,...,n, 


with {a; (0), b;(0), c:(@), di(0)}4 given by the following expressions: 


C A Ci 3 ra 3 
i i Pi Si i Pj Si i 
a; (0) = burns — ids + fi (-msi4i+ i) 4 = (« 4), 
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fiat = fi qe 30741 ie 3p; 
i(9) = ———— - + | -A; + —— —— | —-A; -), 7 
bi(@) a é + ZA oe - i (73) 
1 Pe ‘ 
cj (8) = aA (pis ” — pi a) , (74) 
fen 
ai(0) =~. (75) 
where 
A; = Pit — Pi (76) 


and f,” denotes the second derivative of f(p, 0) with respect to p evaluated at p;, 
1.€., 


n _ PF (.9) 


7 ae ob See hs (77) 


P=Pi 


It is straightforward to establish the identity 


Pi+l fy | ; i Pi+l ah(r) 
~ d r= “Gn _ 50) 4 “tgp -1<p<1, (78) 
plo T—?p P—Pi+1 P—pi Jp T—P 


Employing this identity for the function a (p, 8) and recalling Eqs. (69) and (70), 
we find 


a n—-1 : af r0 
R) 1 £09) 5. YP af) 


dr. (79) 
dp J-1 r—p ie =P 
Equation (71) implies 
Pi-1 af (r,8) Pi+1 Pitt rd 
p _ or dp = nd ar + 2618) rar 
p =P pi =~? op TP 
Pitl 72q 
+340p (80) 
Pi r—p 


The integrals appearing in this equation can be evaluated by employing the 
following identities: 


Pi+1 dr 
= I;(p), (81) 
Pi edo 
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Pi+1 dr 

) = A; + pli(p), (82) 
7 r ~~ p 
Pitl 2dr 1 

) = =(p741—p;) + pAi + pi (p), (83) 
; r-p 2 

where 
ome pane . (84) 
Pi — P 


Using Eqs. (80)-(84), Eq. (79) becomes 


Lf : = Pi+1— P 
dr = C(0)+3 d;(0)A; } p+ D;(p) In , 
i =i = 2 ‘ y pi — p 
(85) 
where C(@) and {D;(p, ayy are defined by the equations 
n—-1 3 
c@= E (0); + 3s(0) (6244 - #)] . (86) 
i=1 
Dj(p, 0) = b;(0) + 2¢;(0)p + 3d;(0)p”, Pi <P S Pi4+1, ial es 
(87) 


After simplifying Eq. (85), we find 


aF(o.0) 1 Toy oe 
ap a {CO +5 ( no 1") P+ Dn—1(p, 8) In |p — pal — Dip, 8) 


n—2 
In |p — pi| + [Di (p, 8) — Di+1(e, 9] In |p — i+ 


i=l] 


1<p<1,0<0<2n. (88) 


In order to eliminate the logarithmic singularities at p = {p,}/, we impose the 
following n equations: 


Dj (oi41,9) = Di+i(oi41,8), G=1,...,.n-2, O<O0<2n, (89) 
Dj (1,8) = Dn—-1(fn, 8) = 0. (90) 


In summary, the inverse Radon transform of a function 7 (p, 0) approximated by 
the cubic spline expression (71) can be written in the form 
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S (x1, x2) = 


An? 


In |e — pn| — Di(p, ) In|p — pi| 


2a 1 " x 
i }c(0) + n= 1") P+ Drip. 0) 


n—2 


+ S> Dil. 0) — Di+ilp, oninie— aul dé, (91) 


i=! 


where C(6) and iD are defined by Egqs.(86) and (87) in terms of 


{b;(8), ¢; (0), d;(0)}%, which are defined by Eqs. (73)-(75) via { fr and fe Itis 
assumed that the functions \ £ (a) are given, whereas \ ih ; (at can be computed 
in terms of i (6) by solving Eqs. (89) and (90). 

In the construction of the so-called “natural” splines, one requires continuity of 
the first derivative, as well as the conditions i i ra ’ = 0. The former requirement 
implies Eq. (89), which eliminates the logarithmic singularities at the interior points 
p=pi, i=2,...,n—1. Inorder to eliminate the logarithmic singularities at the 
end points p; = —1 and p, = 1, we impose Eq. (90) (instead of i = fil = 0). 
In this way, we construct a set of splines “custom-made” for the evaluation of the 
Hilbert transform. 

For a discrete number N of projection angles 0, Eq. (68) yields 


N-1 
1 2nj\ 1 1 
Cte) eee = (x13, =) + 5G(x1, 2,0) + 5G, x2, 2m)p, 


(92) 
where G(x, x2, 0) denotes the evaluation of the RHS of Eq. (88) at p = x2 cos @ — 
x, sin 0. 


Comparison between FBP and SRT for PET 


FBP is reviewed in several publications; see, for example, [21-27]. Here, we only 
note that the inverse Radon transform implemented via the FBP algorithm can be 
expressed by the following formula [28]: 


1 N-1 
(m1, %2) = 5 YF [S (Ep. n) x H(Ep)] (93) 


n=0 


where F—' denotes the inverse Fourier transform in the variable and S(&,, ) is 
the sinogram in the spatial frequency domain given by the expression 


S(E,0) = FEF (p, 3, (94) 
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Fig. 3 The product of the 0.5 
ramp function with typical —— Ram-Lak (Ramp) 
filters used with FBP. The —— Shepp-Logan 
cutoff frequency &, is set at a 0.4 5 — Cosine 
: =} 
0.5 cycles/pixel z == Hanining 
om 0.37) — Hann 
iJ 
= 
S 
a 
o 
© 0.1 4 
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Fig. 4 The two phantoms used for the simulation studies 


with F denoting the direct Fourier transform. The function H(&,) appearing in 
Eq. (93) denotes the product of the ramp function || by some appropriate filter 
function. Figure 3 depicts H(&,) for some commonly used filters in the FBP 
reconstruction algorithm. 

In what follows, we compare FBP (with ramp filter) with SRT using certain well- 
known “quality measures.” These rigorous comparisons appear to indicate that SRT 
has certain advantages. Further comparisons can be found in [9] and [7]. 


Simulated Data 
We will employ two well-known phantoms, namely, an image quality (IQ) phantom 
and a slice of a digital 3D Hoffman phantom [29]. These two phantoms are displayed 
in Fig. 4. 

The IQ phantom, which simulates the human torso, will be used in order to 
establish how well each algorithm can determine hot and cold lesions of variable 
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size inside a warm background. It consists of two circular cold regions (with 
diameters of 38 and 32mm) and four circular hot regions (with diameters of 25, 
19, 15, and 12mm) inside a larger warm region that simulates the background. 
The radioactive concentration ratio (RCR) between hot regions and the surrounding 
warm background is 4:1 for the three hot regions. 

The Hoffman phantom simulates a cerebral PET study. It contains a complicated 
radioactivity distribution within small anatomical features; thus, it allows the 
investigation of the performance of each algorithm in a more realistic situation. The 
Hoffman phantom contains three distinct radioactive regions: gray matter (GM), 
white matter (WM), and cerebrovascular fluid (CSF). The RCR between GM and 
WM is 5:1. The radioactivity concentration in the CSF region is zero. 

The GE Discovery ST PET scanner is simulated by employing STIR. Details of 
the scanner can be found elsewhere [30]. The two phantoms are placed in the center 
of the imaging system, and 2D projections are generated in STIR using a ray-tracing 
technique. Scatter and attenuation are not modeled. The relevant sinograms provide 
the noiseless PET measurements. For each noiseless sinogram, 20 Poisson noise 
realizations are generated at 5 different levels (NLI—NL5), where NLS corresponds 
to the highest noise level applied. 

The SRT algorithm constructs an image in a raster scan format, by scan- 
ning all pixel locations (x;,x2) and then calculating the integral over 6 of 
the derivative of the half-Hilbert transform, which is approximated by Eq. (88). 
In the important case that the boundary of the object being imaged is con- 
vex, a pixel which is outside the object and hence has zero value, can be 
singled out from the sinogram by first identifying the detector locations for all 
angles @ that receive contribution from this pixel; then, for every (x1, x2), if 
there is even one 6 such that fp. 0) = 0, it follows that f(x1, x2) must be 
zero. Using this condition, we can restrict the reconstruction process only to 
pixels within the object boundary and hence obtain a “cleaner” reconstructed 
image. 

By restricting reconstruction within the object boundaries (described above) and 
by employing certain symmetry conditions (see [7]), it is possible to decrease 
substantially the reconstruction time. The actual time depends on the size of the 
sinogram, the reconstruction grid, and the size and the complexity of the object 
being imaged. For example, for the simulated Hoffman phantom (sinograms of size 
221 x 210 and 221 x 221 reconstruction grid), the reconstruction time is about 
2.1s per sinogram, executed on a PC with Intel® Core™4i7-920 Processor. We 
note that no parallel programming or other accelerating techniques are employed, 
which of course will decrease further the reconstruction time. The corresponding 
reconstruction time for FBP in STIR is about 0.3 s. 

In order to determine the quality of the reconstructed images, the contrast and 
bias for the IQ and Hoffman phantom are calculated. The contrast for the hot and 
cold regions are calculated according to [30]. The bias is defined as the mean 
deviation over all realizations of the mean pixel value within a region of interest 
(ROD from the actual activity concentration, i.e., 
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~~ Xtrue ’ (95) 


iZ 
bias = |abe 


r=1 


where R is the total number of realizations, Xtue is the true activity concentration, 
and X,. is the mean activity concentration within an ROI of realization r with M 
number of pixels; X, is given by 


X,= _ yy x. (96) 


The coefficient of variations (COV) is calculated using the expression 
cov =<, (97) 
m 


where o and m are the standard deviation and the mean of the measured activity in 
the background ROL, averaged over all realizations. 

For the IQ simulated phantom, comparisons between SRT and FBP reconstructed 
images with no noise as well as with noise (NL2, NL4, and NLS) are shown in 
Fig. 5. For economy of presentation, images from NL1 and NL3 are not shown. The 
noisy images presented are representative reconstructions of one realization at the 
specific noise level. Both SRT and FBP can generate negative values in pixels (for 
cases where the value of the original phantom is very low or zero). In all images 
presented, the all-black color corresponds to zero values, whereas the white color 
represents the maximum value of the distribution. 

Although the reconstructions from both methods appear similar, there exist two 
main differences: From visual inspection, it is clear that there exist differences in 
the noise texture between the SRT and the FBP reconstructions. Specifically, the 
reconstructions obtained from SRT appear more noisy than those obtained from FBP 
at every noise level. Furthermore, the SRT reconstructions are completely clear from 
streak artifacts outside the object, whereas some small streak artifacts are present in 
the FBP reconstructions. 

The contrast, Chor, for the two smallest hot spheres of the IQ phantom (15 and 
12mm) as a function of COV is presented in Fig.6. The SRT algorithm exhibits 
higher contrast in all three lesions independently of noise level. This advantage 
in contrast increases as the size of the lesion decreases. Indeed, we observe no 
differences in Cia for the 38-mm and 32-mm cold spheres but a small improvement 
for the SRT images in Cyt for the 25-mm lesion and larger improvement for the 19- 
mm lesion. 

The percent bias generated by the reconstruction algorithms for the two smaller 
hot lesions as a function of noise level is presented in Fig. 6. The bias is negative 
in all cases. There are no considerable differences in bias between SRT and FBP 
for the cold lesion and for the largest hot lesion. The bias had small variations as 
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Fig. 5 Reconstructed images of the simulated IQ phantom at various noise levels (the noise level 
increases moving from left to right): (a) no noise, (b) noise level 2, (c) noise level 4, and (d) noise 
level 5 
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Fig. 6 Contrast and bias vs. COV for the two reconstruction algorithms obtained from the 
reconstructed images of the simulated IQ Phantom. Note that the leftmost data point in each line 
curve corresponds to the noiseless case, while the rightmost data point corresponds to the NLS case 


a function of COV for both SRT and FBP. FBP appears to give a higher bias in all 
cases reaching about 9 % for the 12-mm lesion. The percent bias for the FBP images 
increases as the lesion size decreases. The percent bias for SRT has small variations 
as a function of lesion size. The situation is similar for the 19-mm hot lesion. 

For the Hoffman simulated phantom, comparisons between SRT and FBP 
reconstructed images with no noise as well as with noise (NL2, NL4, and NLS) 
are shown in Fig.7. The noisy images presented are representative reconstructions 
of one realization at the specific noise level. All anatomical features of this phantom 
can clearly be identified by both algorithms for all selected noise levels. Small streak 
artifacts outside the object are present in the FBP reconstructions, whereas the SRT 
reconstruction provides images with no such artifacts. 
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SRT 


FBP 


Fig. 7 Reconstructed images of the simulated Hoffman phantom at various noise levels: (a) no 
noise, (b) noise level 2, (c) noise level 4, and (d) noise level 5 


Contrast plots between GM/CSF and WM/CSF as a function of noise level 
are presented in Fig.8a, b. There exists a small improvement in contrast for 
the SRT algorithm, particularly for the case of the WM. RCR plots comparing 
the two reconstruction algorithms as a function of noise level are presented in 
Fig. 8c. The RCR calculations between GM and WM suggest that FBP slightly 
underestimates the actual RCR value (dotted line), whereas the RCR calculated 
from SRT reconstructions is closer to the actual value. The percent bias for the 
GM as a function of noise level is depicted in Fig. 8d. There is a negative bias in 
both algorithms, similar to the case of the IQ phantom. The bias is 4 % for the FBP 
and about 2 % for the SRT algorithm. The bias for the GM is under 0.8 % for both 
algorithms. 


Real Data 

Real-data acquisitions are performed using a commercial ARGUS-CT small-animal 
PET-CT system (SEDECAL S.A., Madrid, Spain). The PET tomograph of this 
system is identical to the GE Healthcare eXplore VISTA small-animal PET scanner 
which is described elsewhere [31]. 

The following two phantoms are used: (a) a NEMA image quality phantom and 
(b) an in-house Derenzo phantom. A NEMA phantom, designed in accordance to the 
specifications of the NEMA NU 4-2008 quality phantom [32], is employed in order 
to determine the noise and contrast properties of each algorithm. This phantom is 
separated into three main parts: a fillable cylindrical region 30 mm in diameter and 
30 mm in length; a solid region with 5 fillable rods with dimensions of 1, 2,3, 4, and 
5mm each; and a uniform region with two cold region chambers 8 mm in diameter. 
Schematics of the three distinct parts of the NEMA phantom are shown in Fig. 9. 
The entire phantom is filled with 15.8 MBq of '8F aqueous solution, and one of the 
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Fig. 8 Contrast, RCR, and bias vs. COV comparisons between the two reconstruction algorithms 
for the simulated Hoffman phantom. Contrast was determined for GM with respect to CSF (a) and 
for WM with respect to CSF (b). RCR between GM and WM was also determined (c). The dashed 
line indicates the actual RCR between GM and WM of the simulated Hoffman phantom being 
imaged. The bias is presented as a percentage of the true activity concentration of the lesion being 
measured (d). Note that the leftmost data point in each line curve corresponds to the noiseless case, 
while the rightmost data point corresponds to the NL5 case 
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Fig. 9 Schematics of the three distinct parts of the NEMA NU 4-2008 phantom used for the real- 
data studies 


8-mm cold chambers is filled with nonradioactive water while the other one remains 
with air. A 30-min PET scan is acquired in two bed positions. 

An in-house Derenzo phantom is used in order to test the resolution limitations 
of each algorithm. The Derenzo phantom consists of 31 microcapillaries arranged 
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Fig. 10 Reconstructions of three slices of the NEMA NU 4-2008 image quality phantom and a 
slice of the Derenzo phantom, acquired by the ARGUS-CT small-animal PET-CT system: (a) SRT 
with no thresholding, (b) SRT with thresholding, and (c) FBP with a ramp filter. Note that the 
8-mm cold chamber on the left of the NEMA 2 slice is filled with nonradioactive water and the one 
on the right with air 


in six different sectors. The capillaries are separated by 2, 3, 4, 5, 6, and 8mm, 
respectively, and there is no material between them. The phantom is filled with 
5.6 MBg of '8F aqueous solution, and a 60-min PET scan is performed. 
Reconstructed images obtained via SRT and FBP (including SRT without 
sinogram thresholding) of three slices of the NEMA NU 4-2008 image quality 


phantom are presented in Fig. 10. Small streak artifacts are present in both SRT and 
FBP reconstructions. These artifacts are reduced in SRT after applying sinogram 
thresholding. 


COV and contrast calculations using the uniform slice and the cold chamber slice 
of the NEMA phantom are presented in Fig. 1 la, b. The SRT reconstructed images 
exhibit slightly higher COV values in comparison to FBP. The contrast values for 
both the water and air chambers are similar for both algorithms. 

Reconstructed images of the Derenzo phantom are shown in Fig. 10. All recon- 
structed circular sources are clearly visible with both methods. Fig. | 1c illustrates 
the contrast for the various sectors of the Derenzo phantom calculated from the SRT 
and FBP reconstructed images. The contrast in the SRT reconstructed images is 
higher than the contrast of the FBP images; the difference in contrast between SRT 
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Fig. 11 Results from analyzing real NEMA NU 4-2008 and Derenzo phantom images: (a) COV, 
(b) contrast for water to background and air to background, (c) contrast for the different sections 
of the Derenzo phantom, and (d) line profiles obtained along the 2-mm separated capillaries of the 
Derenzo phantom. The dashed lines correspond to the normalized profile values through the actual 
phantom 


and FBP images becomes larger as the center-to-center spacing between the lesions 
of the Derenzo phantom becomes smaller. Figure | 1d illustrates the line profiles 
obtained along the 2-mm separated capillaries of the phantom. SRT resolves the 
2-mm separated holes slightly better than FBP. 


5 SRT for SPECT 


In order to derive the final formula for IART, we first derive the following 
simplification of Eq. (21): 


1 2n M 
F192) = 5 i exp {M(x,2,6)} 1 | — cos t-*29) 
2m Jo Tis 
pig Dieta) G(p, 6) — dG (0) 46, (98) 
Ox, dp 


p=x2 cos O—x, sin 6 
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where the functions M and G are defined by 


M(x. x2,8) = f LL(s cos 0 — psin 8, s sin@ + pcos 0)ds (99) 
‘ as reir 
and 
= i(p, 8) Cc : S 
G(p, 8) = exp) =F! [cos F(p, 8))G“(p, 8) + sin(F(p, ))G(p, 8)] 
(100) 
with F, G°, G* defined by 
1 (% fr, 6 
F(p,0) = =f BED) i (101) 
2% Joo T—p 
1. f (u(r, 6 . d 
G°(p,0) = =¢ exp a cos(F(r, 6) fulr, 0) ——, (102) 
20 Joo 2. r—p 


(0.8) = 5G exp | AO sin. 0)) Flr, 
Jes 2 r-—p 
—00o< p<, 0< 6 <2z. (103) 


In order to derive Eq. (98), we insert the operator 0,, —i0,, inside the integral in the 
RHS of (21) and use the identity 


' Ox ax (ax ax 
(8x, ~ Fs) = Fe + ap i (Fee " 2p) 
= (cos 6 —i sin 6) dt + (— sin —i cos 6) dp = e'° (9, —id,). 
(104) 


Then, Eq. (21) becomes 


(0x, — 105,)J (x1, x2, 0) = cet - wx, x2) Ly fut 


0M 0M A 0 7 
ie Seon) ee 
(sin Dx cos x) | (' hu) dp (i wi) | emis an xy ose 
p=x2 cos O—x, sin@ 


(105) 


The contribution to Eq. (21) of the first term of the RHS of Eq. (105) vanishes. 
Indeed, this contribution equals 
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d0 = 


p=x2 cos O—x, sin@ 


Qn ~ 
— po, x2) [ exp {M(x1,x2, 8)} (Lu f.)(0.8) 
Qn 
-n(aix) f J(x1,X2,0)d0. (106) 
0 


But Eqs. (1.8) and (2.9) of [33] evaluated at 1 = 0 imply 


2n 
i J(x1,%2,9)d0 = 0. 
0 


(107) 


Thus, using Eq.(105) in Eq. (21), we find Eq.(98) provided that the following 


identity is valid: 
(108) 


i (Li fu) (p, 0) = 2G(p, 6). 


In order to derive this equation, we will use the equations 


A 


exp {Pf} = exp} —£ — iF ; exp {P* i} = exp | & iF, 
where F denotes half the Hilbert transform of fi in the variable p; see Eq. (101). 


Hence, 

exp{P~ jt} P~ exp {—P~ a} fu = 
RE wal, ! ae i 

a + iF} Fut re («x 3 + iF fs) (109) 


| 


and 
exp {—P* ji Pt exp {P* i is 


- = | fn) _ (110) 


A 


Bh, 1 a i | 
exp }-5 + iF| Ls) 5 —iF| Sut a («| 
where H denotes the Hilbert transform in the variable p defined in Eq. (11). Using 


the above equations in Eq. (23) and simplifying, we find 


A 


iL, f.)o:0) = exp |—5| x 
[expt-iF} H («x é a iF| fu) + exp {iF} H («| é = iF| fi) far 


It is important to note that the RHS of Eq. (111) is real. 
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Using the definitions (102) and (103), after some simplifications, Eq. (111) 
becomes 


A 


i (et) (p, 8) = exp -5 [cos(F)G© + sin(F)G* | : (112) 


6 Conclusion 


PET and SPECT constitute two important medical imaging techniques. Using 
these techniques, images can be obtained by employing either analytic methods or 
iterative methods. 

Regarding analytical methods, we note that simple models for PET and SPECT 
can be formulated in terms of the inverse Radon transform and the IART, respec- 
tively. FBP provides the most well-known numerical implementation of the inverse 
Radon transform. FBP is currently used commercially for both PET and SPECT, in 
spite of the fact that SPECT involves the [ART and not the inverse Radon transform 
(this is a consequence of the fact that until recently no analytic formula was available 
for the IART). 

Here, we have reviewed a general method for deriving transform pairs and have 
used this method to derive the inverse Radon transform and the IART. Furthermore, 
we have presented a novel numerical technique based on “custom-made” cubic 
splines for the numerical implementation of both the above inverse transforms. 
Rigorous studies comparing this novel technique (called SRT) with FBP for PET 
suggest that SRT has certain advantages. In future clinical studies, efforts will be 
made to delineate this advantage in concrete clinical situations. 

The main advantages of the analytical methods, like FBP and SRT, are speed and 
simplicity. However, in these methods, it is difficult to incorporate complex physical 
phenomena such as attenuation and scatter. In FBP, noise issues are treated by 
selecting appropriate filtering parameters, such as the roll-off and cutoff frequencies 
of the reconstruction filter (usually at the expense of spatial resolution). Another 
disadvantage of FBP is the streak artifacts that are particularly prominent near hot 
regions of the object. For SRT, it is possible to eliminate these effects at least outside 
the main region of interest and hence to obtain a “cleaner” image. 

The predominant iterative algorithms are the maximum likelihood expectation 
maximization (MLEM) algorithm [34] and its accelerated successor the ordered 
subset expectation maximization (OSEM) algorithm [35]. The main advantage 
of the iterative algorithms is the ability to model several aspects of the imaging 
system, including elements of the noise characteristics, sinogram blurring due to 
detector crystal penetration, inter-crystal scatter, depth of interaction, and photon 
attenuation [36,37]. As a consequence, iterative methods can improve image quality 
and achieve considerable resolution recovery. However, iterative algorithms require 
more computing time and power, particularly when details of the physical model are 
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included. Iterative techniques are now in widespread use in clinical and preclinical 
systems. This is due to the speed improvement provided by OSEM and the recent 
computer hardware improvements (processing and storage). 

Most commercial clinical and preclinical PET systems allow the use of either 
FBP or OSEM for image reconstruction. Currently, in OSEM, the main challenge 
is the selection of the proper number of subsets and iterations [38], as well as the 
choice of a suitable post-reconstruction filter (if needed). Stopping the algorithm at 
the proper number of iterations is important, since EM-based algorithms suffer from 
noise/bias trade-off. Stopping the iteration process after convergence is reached 
results in a noisy image, whereas stopping the process too soon results in a less 
noisy image which however is biased toward the image assumed at the initial step. 
In order to resolve this issue, several regularization schemes have been proposed 
[39]. In spite of these improvements, a recent dynamic brain PET study by Reilhac 
et al. [40] concludes that analytical methods are more robust to low-count data than 
iterative methods. Furthermore, the positivity constraint imposed to the sinogram 
and image space by the EM-based reconstruction algorithms leads to overestimation 
(positive bias) of the low-activity regions [40-42]. 

The advantage of analytical methods for low-count data will be increasingly more 
important. Indeed, in the 1980s, medical imaging was responsible for only about 
15 % of the total radiation exposure to US population from all sources; now this 
proportion is 50 % [43]. In 2010, 70 million CT scans were performed in the USA, 
and the radiation dose from CT scans is 100-500 times those from conventional 
radiography. Taking into consideration that CT is now used for screening for lung 
cancer [44] and that the use of PET-CT and SPECT-CT is expanding, it is imperative 
to be able to produce images with the least possible radiation. 
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Abstract 

This survey starts with a brief description of the scientific relevance of electron 
tomography in life sciences followed by a survey of image formation models. 
In the latter, the scattering of electrons against a specimen is modeled by 
the Schrédinger equation, and the image formation model is completed by 
adding a description of the transmission electron microscope optics and detector. 
Electron tomography can then be phrased as an inverse scattering problem and 
attention is now turned to describing mathematical approaches for solving that 
reconstruction problem. This part starts out by explaining challenges associated 
with the aforementioned inverse problem, such as the extremely low signal- 
to-noise ratio in the data and the severe ill-posedness due to incomplete data, 
which naturally brings up the issue of choosing a regularization method for 
reconstruction. Here, the review surveys both methods that have been developed, 
as well as pointing to new promising approaches. Some of the regularization 
methods are also tested on simulated and experimental data. As a final note, 
this is not a traditional mathematical review in the sense that focus here is on 
the application to electron tomography rather than on describing mathematical 
techniques that underly proofs of key theorems. 

Acronyms 

ART Algebraic Reconstruction Technique 

CCD Charged Coupled Device 

CPMV Cowpea Mosaic Virus 

CTF Contrast Transfer Function 

EELS Electron Energy Loss Spectroscopy 

EDS Energy-Dispersive X-ray Spectroscopy 

ET Electron (microscopy) Tomography 

ELT Electron A-Tomography 

FBP Filtered Back-Projection 


HAADF _ High-Angle Annular Dark-Field 
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HRTEM _ High-Resolution TEM 


POCS Projection Onto Convex Sets 

LDDM Large Deformation Diffeomorphic Metric Mapping 
ML Maximum-Likelihood 

ML-EM = Maximum-Likelihood Expectation Maximization 
MTF Modulation Transfer Function 

PSF Point Spread Function 

SART Simultaneous ART 

SEM Scanning Electron Microscope 

SIRT Simultaneous Iterative Reconstruction Technique 
SSC Slow-Scan CCD camera 

STEM Scanning Transmission Electron Microscope 
TEM Transmission Electron Microscope 

TMV Tobacco Mosaic Virus 

TV Total Variation 

WBP Weighted Back-Projection 

WKB Wentzel—Kramers—Brillouin 

1 Introduction 


Imaging is today an essential tool in many different areas of scientific and 
technological research. It is also widely used in investigations in fields as diverse 
as arts and jurisprudence. As such, it is integrated into a variety of devices and 
procedures routinely used in society. A prime example is microscopy that enables 
investigation of objects that are too small for the naked eye at a variety of scales, 
from atomic to sub-millimeter. Therefore, various forms of microscopy are crucial 
in industrial processes, in life sciences, in comparative studies of chemistry and 
geology, and in diagnostic medicine. 

Traditional imaging techniques provide 2D images whose usage involves signifi- 
cant portion of interpretation since the object/phenomena under investigation almost 
always takes place in three spatial dimensions, plus time if temporal variation is 
included. 3D images are therefore to be preferred whenever possible. This applies in 
particular to life sciences and drug discovery, where a central topic is to understand 
the machinery within the cell responsible for supporting life and disease. An 
important part in this quest is to map the spatial and temporal arrangement of the 
molecules engaged in this machinery. The structure determination problem, which 
seeks to recover the three-dimensional structure of an individual molecule at highest 
possible resolution in its natural environment, has therefore come to play a central 
role. 

Addressing the structure determination problem is, not surprisingly, both exper- 
imentally and computationally very challenging. X-ray crystallography and nuclear 
magnetic resonance are two established approaches, but these cannot recover the 
structure of an individual molecule in its natural environment within the cell. 
Electron microscopy offers alternative means to study macromolecules, molecular 
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complexes and supramolecular assemblies in 3D. Three approaches have been 
developed, electron crystallography [68], single particle analysis [61], and ET. 

This review focuses on ET, which is the only method for 3D structural studies of 
individual assemblies and subcellular structures in their natural environment. The 
idea in ET is to recover the structure of a specimen from a series of micrographs, 
which are 2D TEM images, using principles of tomography. Since its introduction 
to the scientific community in 1968 [41, 85, 179], there has been tremendous 
progress in instrumentation and sample preparation that has resolved many of the 
difficulties associated with data collection. Most of the computational challenges 
however remain to be addressed. Established approaches for image reconstruction 
are direct adaptations of methods from medical imaging, used with inadequate 
appreciation for the finer points of mathematics. Therefore, 3D reconstructions 
have a resolution that is reliable only for studying cellular substructures, rather 
than studying individual small molecules (“small” in this context refers to protein 
molecules of size smaller than 200 kDa). This in turn has severely limited the 
usefulness of ET in life sciences. 

Another rapidly growing application area for ET is 3D characterization and 
metrology of nano-structures in material sciences and semiconductor manufactur- 
ing. In these applications, data is often acquired using a variety of different TEM 
imaging modes (paragraph on p.941), see [126, 127, 181] for a nice review. The 
focus in this review is on life sciences, even though we occasionally mention 
material science applications. 


2 The Transmission Electron Microscope (TEM) 


The starting point of electron microscopy can be attributed to Hans Busch who 
laid the theoretical basis for the electron microscope and designed the first working 
electron lens [23,24]. Ten years later, based on the work of Busch, Ernst Ruska 
constructed the first operational TEM [98]. Ruska was later awarded the 1986 Nobel 
Prize in physics. 

The operational principle of an electron microscope is similar to a light micro- 
scope (Fig. 1). A source emits electrons that are accelerated (typically by 200 kV) 
on their way to the specimen. A condenser system controls how the specimen is 
illuminated. After passing through the electron—transparent specimen, an electron 
optical system collects the electrons of interest and directs them onto the image 
plane, where an intensity is generated. This is detected and converted to a gray- 
scale image (micrograph) by means of a detector system. The beam electrons scatter 
elastically and/or inelastically against the atoms in the specimen. This interaction 
can generate a range of secondary signals whose analysis reveal different properties 
of the specimen. 

A modern TEM can be operated under different settings (imaging modes) to 
image some of the aforementioned signals. Each imaging mode is determined by 
(1) how the specimen is illuminated, (2) which electrons that are selected to take part 
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Electron source/gun 


=— Condenser system 


Specimen (thin) 


= Objective lens 
‘— Projector lenses 


Fig. 1 A modern TEM with its column exposed. The electron gun with the source is at the top. 
Electrons travel downwards, passing through the condenser system before they scatter against the 
specimen. Scattered electrons pass through the optics (objective and projector lenses) and form an 
image at the bottom (image courtesy of FEI) 


in the image formation and where they are registered, and (3) the type of detector. As 
an example, dark-field mode uses only the deflected electrons for image formation, 
whereas in bright-field mode only the undeflected electrons take part in the image 
formation. 

ET in life sciences is mostly based on conventional bright-field TEM imaging. 
The specimen is here illuminated by a parallel beam of incident electrons (uniform 
illumination mode), and transmitted electrons that pass through the aperture in the 
back focal plane of the objective lens form an image in the image plane. The aperture 
is centered (bright-field mode), so only electrons that scatter with very small angle 
take part in the image formation. 


TEM Imaging Modes 

The specimen can be illuminated by focusing the electrons onto a spot (scanning 
mode), which leads to Scanning Transmission Electron Microscope (STEM). Next, 
the objective aperture can be placed as to block the direct beam of un-scattered 
electrons, while one or more diffracted beams are allowed to pass (dark-field mode). 
Finally, instead of registering the image in the image plane, it can be registered 
in the back-focal plane of the objective lens (diffraction contrast mode), which 
corresponds to imaging the diffraction pattern of the specimen. 
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A common mode is selected area electron diffraction that uses uniform illumina- 
tion with diffraction contrast imaging, suitable for investigating spatial variation 
of diffraction properties in a (crystalline) specimen. Convergent-beam electron 
diffraction is the corresponding set-up based on scanning mode illumination. 
Another mode is High-Angle Annular Dark-Field (HAADF) that uses scanning 
mode illumination and micrographs are acquired in dark-field mode. The electrons 
that take part in image formation are incoherent elastically scattered electrons, 
so contrast depends strongly on the average atomic number of the scatterer 
(“Z”-contrast). Finally we mention analytical TEM that uses energy resolving 
detectors, Energy-Dispersive X-ray Spectroscopy (EDS) or Electron Energy Loss 
Spectroscopy (EELS), combined with scanning mode for illumination. This allows 
one to acquire a “chemical map” of the specimen. An EDS spectrometer does this 
by measuring the X-rays emitted from the specimen, whereas an EELS spectrometer 
measures those transmitted electrons that undergo a pre-specified energy loss. 

The above modes are mostly used in material sciences and semiconductor 
industry, even though there are usages in life sciences. The interested reader may 
consult [63] and [7, chapter 3] for a more detailed account. 


Sample Preparation 


Samples to be imaged in an electron microscope generally require processing to 
produce a suitable specimen. Sample preparation techniques depend on the sample, 
the information one seeks to image, and the type of microscope. A mathematician 
may safely ignore most issues related to sample preparation, but some aspects of 
it that do have algorithmic implications. Before we discuss these (in paragraph 
on p. 943), we provide a very brief overview of sample preparation. 


Overview 

In general, all sample preparation techniques include some kind of fixation that aims 
at solidifying the specimen. This is needed because electron microscopy imaging 
is performed under high vacuum conditions. Next, thinning might be required to 
ensure that the specimen is thin enough so that electrons can pass through. Both 
these steps are done while preserving as much as possible of the structural integrity 
of the specimen. Finally, a user might choose to add substrates into the specimen in 
order to increase contrast (contrast enhancement). A wide range of techniques are 
now available, see [7,8] for a comprehensive review. 

In life sciences, the preferred fixation technique is by freezing (cryo-electron 
microscopy) [148, section 3.1]. If the specimen is liquid, or thin enough (<10 wm), 
then one can use plunge-freezing. Here the specimen is applied to a support and 
then dropped into a cryogen (such as liquid ethane at less than —160 °C) where 
sufficiently fast cooling speeds are reached that prevent the formation of crystalline 
ice. This solidifies the specimen with minimal artifacts. It is the preferred cryo- 
fixation method for in vitro specimens, i.e., specimens that contain free particles 
(proteins and macromolecular assemblies) in an aqueous solution. Samples that 
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are too thick for plunge-freezing are treated with high-pressure freezing, which 
combines rapid temperature decrease with the application of high pressure that 
lowers the melting point of water. It allows vitrification of specimens up to 200 jum 
in thickness. 


Relevance to Mathematics 

Let us first consider usage of contrast enhancement, like in negative staining. The 
high atomic numbers for contrast agents improve the signal-to-noise properties 
of the micrographs, so noise becomes less of a problem. On the other hand, 
contrast agents have difficulties reaching the interior of the molecular assemblies, 
so micrographs show outlines of the molecules rather than their internal structure. 
Therefore, contrast enhancement is reserved for low- or medium-resolution TEM 
imaging. In the context of ET, we can in the forward model often disregard phase 
contrast (section “Forward Operator for Amplitude Contrast Only”). Next, since 
focus is on determining the 3D shape of the sub-structures within the specimen, 
rather than their internal structure, 3D reconstruction methods should preserve edge 
information whereas smooth gray scale variations are of less interest. The impact 
of local data (p. 974) can also be quite severe in this setting (section on p. 985), so 
if possible, one should use reconstruction methods that are less sensitive to such 
issues. 

Cryo-fixated unstained specimens are on the other hand thin and weakly scatter- 
ing. ET of High-Resolution TEM (HRTEM) imaging data from such specimens 
must model phase contrast. Specimens that contain particles in aqueous buffer 
(in vitro specimens) are fixated by plunge freezing. These are imaged using a very 
low electron dose, so micrographs are very noisy and 3D reconstruction methods 
must efficiently utilize whatever a priori information that is available, like regularity 
or sparsity adapted to this setting. Impact from local data is on the other hand less 
severe since most of the specimen is vitrified aqueous buffer. 

Specimens that are derived from tissue (in situ specimens) typically result 
in micrographs with much better signal-to-noise ratio (although not as good as 
negatively stained specimens). The background will have significant structure, so 
regularity assumptions, like sparsity, will have to be quite different from those used 
for in vitro specimens. The impact from local data is also more severe as compared 
to imaging data from in vitro specimens. 


3 Basic Notation and Definitions 


R and C denote the fields of real and complex numbers, R+ and Cy are positive 
numbers (for complex numbers, this means positive real and imaginary parts), and 
IR” denotes the n-dimensional vector space over R. Next, for x € R? we let xt 
denote the hyperplane in R? that is orthogonal to x,i.e.,.x+ := {y © Ro: x-y = 
0}. S? C R? is the 3D sphere, and for @ € S* and p € R, we let pw + w+ denote 
the hyperplane of points x = pw + y where y € w+. 
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Next, F and « denote the Fourier transform and convolution of functions defined 
on R". Furthermore, given a function U: R* — C that decreases sufficiently fast at 
infinity, say U € .Y(82,C), its ray transform P(U) (also called projection in the 
ET community) is defined as 


Co 
PUU(w.x) = | U(x +s@)ds forw € S* andx eo (1) 
—oo 

More precise mathematical conditions that are required for the existence of the ray 
transform are stated in [122, section 3.5]. Finally, following [136, eqs. (2.29) and 
(2.30)], for fixed @ € S? we let F,,1 and ®,.1 denote the corresponding Fourier 
transform and convolution on the (two-dimensional) hyperplane w+: 


F,s[U] (&) := fe U(x)ei*'Fdx foré € wt, 


(U ® H)(x) = 


U(x —y)H(y)dy forx e€ ee 


4 The Forward Model 


This section gives an overview of TEM image formation models (forward models) 
relevant for ET. It includes a short derivation of various models with discussions on 
their validity and computational feasibility. Much of the material is therefore physics 
related, so a reader primarily interested in image reconstruction could skip this part 
and directly consult (57) (or section “Forward Operator for Combined Phase and 
Amplitude Contrast” for a slightly more detailed summary). 


Basic Assumptions 

We only consider conventional bright-field TEM imaging of amorphous specimens, 
since this is the prevalent imaging mode for ET in life sciences. The starting point 
is to assume that the imaging electron and the specimen form a closed system, i.e., 
there is no interaction with the environment. Next, we also assume that successive 
imaging electrons can be treated independently and any interaction between them 
can be neglected (independent electron assumption). This holds under typical TEM 
imaging conditions in ET where the mean separation between two successive 
electrons is much larger than the specimen thickness and the length of the electron 
wave packet [63, p. 85]. Hence, wave mechanical notions, like “interference,” refer 
to the wave crests of an individual electron (i.e., self-interference). 

Bearing in mind these two basic assumptions, an ideal model would be based 
on solving the Schrédinger equation for the electron microscope (including the 
specimen) as a whole. This is clearly unfeasible, so the first step is to separate 
the problem into four parts: (1) illumination, (2) electron—specimen interaction, 
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(3) optics, and (4) detection. A great number of approaches have been developed 
for modeling each of the above parts. 


Contrast Mechanisms 
Models for TEM imaging fall into two categories depending on how image contrast 
is related to electron—specimen interaction and optics. 

Amplitude contrast models (section “Forward Operator for Amplitude Contrast 
Only’) assume all contrast variations in a micrograph are due to the removal of 
electrons from then beam (amplitude contrast). This is adequate at medium and low 
resolution. At higher resolution, like in HRTEM where image features of interest 
are smaller than the coherence length of the electron, one also needs to account for 
the phase shift that an electron undergoes as it scatters against the specimen (phase 
contrast). Combined phase and amplitude contrast models, that are briefly described 
below, account for both the amplitude and phase contrast. 

The starting point for the combined phase and amplitude contrast model is to 
assume perfect coherent imaging where incident electron waves are monochromatic 
plane waves (uniform coherent illumination) and electrons scatter only elastically. 
This leads to a stationary scattering problem where the specimen remains in the 
same quantum state with scattering properties fully described by its electrostatic 
potential. The treatment of the electron is quantum mechanical. The electron— 
specimen interaction is then modeled by the scalar Schrédinger equation and 
the picture is completed by adding a description of the effects of the optics 
and the detector of the TEM, both modeled as convolution operators. Inelastic 
scattering and incoherent illumination introduce partial incoherence, so the basic 
assumption of perfect coherent imaging must be relaxed. Inelastic scattering can be 
accounted for within the coherent framework by introducing an imaginary part to 
the scattering potential (absorption potential). Incoherence from the illumination is 
usually accounted for by modifying the convolution kernel that models the optics. 
Finally, linearizing the scattering model and the intensity yields an explicit linear 
relation between the measured intensity and the scattering potential. 


Illumination 


This is where incident electrons are generated (source/gun) and controlled (con- 
denser). 


Basic Components 

Commercially available TEMs either use a thermionic or a field-emission (electron) 
source. The first produces electrons by heating the source, whereas the second 
produces electrons by applying a large electric potential between an anode and the 
source. The performance of the source is characterized by its brightness, temporal 
coherency, energy spread, spatial coherency, and stability. The brightness (the 
current density per unit solid angle of the source) is the most important parameter 
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and it influences the resolution, contrast, and signal-to-noise capabilities of the 
microscope. A source with smaller size gives higher brightness and better spatial 
coherency, but is also less stable. 

The source is incorporated into a gun that can control and focus the electrons. 
According to the type of the source, we have either thermionic gun or field-emission 
gun. The size of the first one is bigger, so it can illuminate large areas on the 
specimen at relatively low image magnification (<50—100,000x) without losing 
current density. The field emission gun has become the preferable electron source 
for HRTEM imaging (for both uniform and scanning mode illumination) due to its 
high brightness and small source size. More information can be found in standard 
text books on electron microscopy, see, e.g., [80, sections 43-50]. 

After leaving the gun, the electrons pass through a condenser system that 
typically consists of two condenser lenses and an aperture. This is the second most 
important part of the TEM, the objective lens (section “Optics”) being the most 
important, since it determines how the specimen is illuminated. One can illuminate 
the specimen with a wide collimated parallel beam (uniform illumination) or with a 
parallel narrow beam (scanning mode illumination). 


Model for Illumination 

Since we consider conventional TEM, micrographs are acquired using uniform 
illumination mode. Furthermore, we will not directly model the source, gun, or 
condenser. Instead, as a starting point, we assume that an incident electron leaving 
the condenser is a monochromatic plane wave traveling along a fixed direction 
@ € S* that is parallel to the TEM optical axis: 


W(x,t) = exp(—itE/h) exp(ikx -@). 


Here, F is the energy of the electron, f is the reduced Planck constant, and k is the 
wave number of the electron, so k = 27/A where A is the wavelength. 

In reality, however, the illumination is only partially coherent. The effective size 
at the electron source is small but not point-like, and the energy spread is narrow 
but still appreciable. A perfectly coherent source would require vanishingly small 
source and negligible energy spread. The effects of such partial coherence is usually 
modeled by a perturbation procedure and incorporated into the TEM optics PSF, as 
outlined in paragraph on p. 961. 


Electron-Specimen Interaction 


This is the model for the scattering of an electron against the atoms in the specimen. 
In the general setting, it is a fairly complex process involving a variety of phenomena 
depending on the energy of the electron, the atomic composition of the specimen, 
and the type of chemical bindings between the atoms in the specimen. An accurate 
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and complete model for all electron scattering phenomena, that includes both 
quantum and relativistic aspects, requires the theory of quantum electrodynamics. 
A slightly less complete framework is based on the time-dependent multi-body 
Schrédinger equation where one approximately accounts for relativistic effects. 
Here, in the full model, the specimen is a quantum-mechanical many-body system 
whose scattering properties are accounted for by the corresponding Hamiltonian. 
This leads to a computationally unfeasible model, so we need further simplifica- 
tions. One is to considering elastic scattering. 


Elastic Scattering 

Elastic scattering refers to the case when there is no (or negligible) transfer or 
energy from the electron to the specimen. This allows us to treat the electron— 
specimen interaction as a two-body problem where the specimen is described 
by its electrostatic potential and the electron is represented by its wave function 
W: IR? x R = C, and the aforementioned multi-body Schrédinger equation for the 
system simplifies to a one-body Schrédinger equation [146, section 1.5]: 


A b= —| ~ihV +eA(x)] Vex. t) —eU(x) W(x, t). (2) 
ot 2m 


In the above, m is the rest mass of the electron, e the absolute value of the elementary 
charge, U: R* — R? is the electrostatic potential, and A: IR? — R? is the magnetic 
vector potential for the stationary electromagnetic field surrounding the system. 


Stationary Model 
Since the electrostatic potential U in (2) does not depend on time, it is sufficient to 
consider standing wave solutions to (2): 


W(x,t) = exp(-itE/h)w(x). (3) 


In the above, y:R* -> C is the complex valued space dependent part of W, 
henceforth called the electron wave. Inserting (3) into (2), and canceling the 
common factor exp(—itE/h) that appears throughout, results in a partial differ- 
ential equation (4a) (stationary Schrédinger equation) for the electron wave w. 
Furthermore, in TEM imaging the specimen is typically “slab-like,’ meaning that 
its extension along the TEM optical axis is much smaller than its extension in the 
plane orthogonal to the optical axis. A natural model is therefore to assume U is 
supported in an infinite slab 2 C R?> that is bounded by two parallel hyperplanes 
Tin and Tout. 

To summarize, the electron—specimen interaction is modeled by the stationary 
Schrédinger equation with a Dirichlet boundary condition on Ij, given by the 
incident electron wave and a boundary condition at infinity (Sommerfeld radiation 
condition): 
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Ey(x) = ~ [inv +ea(x)] y —eU(x)W(x) forx€2, (4a) 


2m 
w(x) =exp(ikx-w) forx € Ip. (4b) 
tim r[ n, (x): V Viel) — ikYse() =0 for|x|=r. (4c) 


Here, Ws in (4c) is the scattered part of y, i.e., W(x) = exp(ikx -@) + Wec(x) and 
n, (x) is the outwards normal at x to the sphere with radii r. Note also that (2 is not 
necessarily orthogonal to the TEM optical axis. 

A further simplification is when scattering takes place in a “‘field-free” region, 
ie.,curlA = B = 0in Q. Due to the invariance under Gauge transformations, this 
implies that A = 0 in 2. Combined with E = h?k*/2m, transforms (4a) into an 
equation of Helmholtz type: 


(A +k?) W(x) = — TUE )W(x) forx € 2. (5) 


The Scattering Operator 
The scattering operator 7° maps a potential U to the electron wave on the 
specimen exit plane Jut, Le., 


Te (U) := v| Ton where wy solves (4a)—(4c). (6) 


Uniqueness and Stability 

The scattering operator in (6) is well defined only when (4a)—(4c) has a unique 
solution. This follows from existence and uniqueness for the magnetic Schrédinger 
equation (4a) in a half space [149, theorem 2.16]. 


Remark I. Formally, the uniqueness result [149, theorem 2.16] is for the case when 
® is orthogonal to Iy:. The proof however extends directly to the case when w = 
Wo on Ij, where Wo € Y?(H,C). 


Remark 2. Standard uniqueness results for the direct scattering problem, like in [36, 
p. 16], assume {2 is bounded and Dirichlet condition is on all of the boundary of 92. 
In our scattering problem (4a)—(4c), 2 is unbounded and we only have Dirichlet 
conditions on a part of the boundary. 


Another important aspect is stability for the direct scattering problem (4a)—(4c), 
which together with uniqueness implies well-posedness. The author is unaware of 
any stability results that are directly applicable to (4a)—(4c). On the other hand, 
[87] provides analytical evidence of increasing stability of the Cauchy problem for 
the Helmholtz equation (5) for increasing wavenumber k. Since TEM imaging is 
associated with high wave numbers, it is reasonable to conjecture that (4a)—(4c) has 
good stability properties. 
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Relativistic Corrections 

The Schrédinger equation (2) is valid only for non-relativistic electron motion. In 
practice, this means kinetic energies should not exceed 10 keV. In TEM imaging, 
electron energies are between 100 and 300 keV, so one must consider relativistic 
wave equations. 

A relativistically correct framework is offered by Dirac’s equation that accurately 
models electron interaction and propagation in practically all electron optical 
systems. The computational complexity associated with using Dirac’s equation has 
however limited its usefulness. An alternative approach is to use relativistically 
corrected quantities in (2) [146, section 1.9]. This is justified from a pragmatic point 
of view if since these corrections provide a model that has a good agreement with 
experimental results. 


Remark 3. The Pauli approximation, which holds for TEM imaging in ET, allows 
one to reduce Dirac’s equation to a scalar relativistic wave equation of the same 
type as (2) [79, section 56.3]. This introduces relativistic corrections into (2) in a 
way that has stronger theoretical justification. 


Inelastic Scattering 

Inelastic electron scattering is characterized by transfer of energy from the incident 
electron to the specimen, causing the latter to change state. For many TEM imaging 
modes, including conventional bright-field TEM imaging, inelastic scattering man- 
ifests itself as a blurring superimposed onto the image generated by the elastically 
scattered electrons [100]. Therefore, if possible, energy filters are used to remove 
inelastically scattered electrons. This however does not completely resolve the issue 
of inelastic scattering, since energy filtered micrographs have contributions from 
inelastic scattering events associated with energy losses too small to be filtered (<1- 
5 eV) [45]. 


The Absorption Potential 

In a rigorous quantum mechanical model for inelastic electron scattering, the 
quantum state of the specimen is changed. Formally, this requires a time-dependent 
scattering model with a wave function and an interaction potential that depends 
on all internal degrees of freedom of the system (specimen and electron). This 
is clearly computationally unfeasible. An alternative approach, that is frequently 
used for modeling TEM imaging of amorphous specimens at 100-300 keV, is to 
approximately account for inelastic scattering within the framework offered by the 
one-body Schrédinger equation (2). 

The idea is to split the aforementioned interaction potential into two parts, one 
time-dependent and the other time-independent [146, p.3]. The time-independent 
part is obtained by taking the time-average of the interaction potential and it models 
elastic scattering, whereas the time-dependent part models inelastic scattering. 
Inelastic scattering can now be accounted for using a first-order perturbation 
analysis w.r.t. time as it tends to infinity. This gives rise to the appearance of 
an additive imaginary part of the time-averaged potential (absorption (optical) 
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potential) [146, p.3], thus formally replacing the real valued function electrostatic 
potential in (2) with a complex valued one, i.e., U = Uze +i Uim where U;e: R?>R 
is the electrostatic potential and Uim: R*? — R is the aforementioned absorption 
potential. 


Interpretation of the Absorption Potential 

The electrostatic potential U,. models the phase shift that an electron undergoes as 
it scatters against the specimen (phase contrast), whereas the absorption potential 
Uim models the decrease in the flux, due to inelastic scattering, of the non-scattered 
and elastically scattered electrons (see [146, p.5 and section 1.10] for a definition of 
the notion of flux in wave mechanics). Hence, the absorption potential accounts for 
amplitude contrast due to inelastic scattering. A more detailed quantum mechanical 
interpretation of the absorption potential is provided in [128][146, section 1.8 and 
chapter 13] [176, section 5.9]. 

Now, any computational treatment of TEM imaging data where phase contrast 
is important, like HRTEM data, has to address the phase problem (section “Phase 
Retrieval”). In many applications, like for ET in life sciences, the specimen is 
viewed only once from a fixed direction. Resolving the phase problem in such cases 
requires one to utilize further knowledge about the complex valued potential U. 
Now, in the context of TEM imaging in life sciences, we argue below that U,. and 
Uim can be considered approximately proportional. This is enough for resolving the 
phase problem in the context of ET. 

Most inelastic scattering events in conventional bright-field TEM imaging are 
either due to plasmon excitation or atomic inner-shell ionization (atom core losses), 
where the former is considered more dominant. Next, inelastic scattering events are 
more frequent for scattering against light atoms, they are almost always incoherent 
and associated with low scattering angles, and their likelihood increases with 
specimen thickness. To summarize, plasmon excitation from light atoms constitute 
the main contribution to the inelastic scattering that damps the interpretable phase 
contrast signal. For TEM imaging of amorphous biological specimens, the light 
atoms mostly make up the background, which for cryo-fixated specimens would 
be the embedding medium, i.e., the main contribution to inelastic scattering comes 
from the vitrified aqueous buffer. Now, the imaginary part of the potential associated 
with plasmon excitation for an amorphous embedding medium, like vitreous ice, is 
expressible by the inelastic mean free path Aine: 


1 me 
where o := 


Uin = —. 
20 Ainel kh2 


This forms the basis for a common assumption in ET, namely that the real and 
imaginary parts Ur. and Ui of U in (2) are proportional to the path length of the 
electron through the specimen. Stated more precisely, there is a function 0: S? > R 
such that 


PUim)(@,x) + O(@)P(Ue)(@, x) forx € wt andw € S’. (7) 


Mathematics of Electron Tomography 951 


Here, ? is the ray transform in (1) and @ is the direction of the incident electron. A 
common further simplification is to let Q be a constant (amplitude contrast ratio), 
which then can be estimated from data (section on p. 985). 


Remark 4. Assumptions of the type in (7) that relate the real and imaginary parts 
of U are known as the “Pagnini approach” in X-ray phase contrast tomography 
community [139]. 


More Accurate Quantum Mechanical Models 

A fundamental difference between modeling inelastic, as opposed to elastic, 
scattering is that the former requires knowledge of the local scattering and excitation 
properties of the specimen. The scattering properties of the specimen are no longer 
fully described by a single real-valued function (the electrostatic potential). In 
general one would need to specify the elements of the transition matrix that encode 
the probability that a specimen undergoes a specific transition from one excited state 
to another. This situation can be somewhat simplified, under certain conditions it is, 
e.g., possible to express the inelastic cross section without explicit knowledge of 
these excited states [79, section 68.6]. It is also clear that a computationally feasible 
approach cannot be based on migrating a wave function that depends on the spatial 
location of all subatomic particles in the specimen. 

A theoretically stringent, yet computationally feasible, framework is based 
on migrating the mutual coherence function instead of the wave function, the 
former being the least amount of information needed to describe interference. The 
scattering properties of the specimen are then modeled by the mixed dynamic 
form factor that accounts for the interference in Fourier space of different scattered 
partial electron waves by their mutual coherence [160]. Approaches based on this 
framework are, e.g., described in [107, 129, 183]. Unfortunately, these approaches 
are still computationally unfeasible in the setting when the specimen is amorphous 
and/or thicker than a few atomic layers, thus limiting their usefulness in ET in life 
sciences. 


Importance of Inelastic Scattering 

The absorption potential model outlined above is adequate for modeling most 
conventional bright-field HRTEM imaging problems with amorphous specimens. 
Combined with (7) and geometric optics approximations (section on p. 954), one 
gets a computationally feasible model for the electron—specimen interaction in the 
context of ET. 

The validity of this image formation model is less clear for TEM imaging of 
specimens in material sciences. As an example, the absorption potential approach 
is not sufficiently accurate when imaging crystalline specimens in diffraction mode 
and most of [146, chapters 1 and 7-8] is devoted to presenting various models that 
are more accurate. Another situation that requires more accurate models of inelastic 
scattering is when specimens are imaged using lower voltages. The motivation is 
from material sciences where thin crystalline specimens undergo radiation damage 
when imaged using acceleration voltages beyond 100 kV. For low accelerating 
voltages (~20 kV), most specimens act as strong scatterers. This requires accurate 
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modeling of inelastic scattering, especially when atoms are light as in biological 
specimens. In this context, models based on the mutual coherence function are the 
most promising [107]. 


Properties of the Scattering Potential 

Let 2 denote the set of complex valued functions on R? that can serve as a potential 
U:R?* — C in (2). Elements in 2 have to posses certain regularity, both from a 
mathematical and a physics point-of-view. One such regularity property motivated 
from the physics is that U fulfills the Rollnick condition, which is the case when 


BEBO | 20). 


The Rollnick condition guarantees that the corresponding Hamiltonian is self- 
adjoint and one has asymptotic completeness. The latter means that no matter how 
the wave-packet is made, particles go far apart from each other when time becomes 
large, so the probability of finding them together in an arbitrary finite region of space 
goes to zero. This allows one to go from the time-dependent Schrodinger description 
to the time-independent description characterized by solutions of type (3) [31, 
p. 133]. See also [195] for the other regularity properties associated with physics. 

Next, we claim that U:R* — Cz whenever U € 2%. The positivity of the 
imaginary part follows from the observation that once there is an inelastic scattering 
event, that partial wave can never be considered as un-scattered or inelastically 
scattered (for that it would have to gain energy from the specimen). Next, the real 
part is the electrostatic potential that, at sub-atomic scale, might actually change 
sign. It is made up of two contributions, one “atomic” given by the superposition 
of atomic potentials as if each atom was in isolation, and one from the charge 
redistribution due to the solvent, ions, and molecular interactions. In the context 
of ET, the isolated atom superposition approximation allows one to account for 
the latter contributions within the “atomic” contributions. Hence, the specimen can 
be seen as consisting of isolated neutral atoms, each modeled as a point charge 
representing the nuclei surrounded by a spherical shell. The sphere represents the 
screening imposed by the shell electrons, so the electrostatic potential from the 
electrons cancels the one from the nucleus outside the sphere (the atom is neutral). 
Since the radius of the sphere is much larger than the radius of the nucleus (which 
here is a point charge), the electrostatic potential from the nucleus must dominate 
the electrostatic potential from the shell electrons within the sphere. Hence, the 
electrostatic potential is non-negative everywhere. 

As a final note, one can put further regularity on elements in 2, e.g., 2 C 
(2, C) (rapidly decreasing) and/or 2 Cc BY (2, C) (bounded variation). 


Mathematics of Electron Tomography 953 


Computationally Feasibility 


The Multi-scale Nature of Electron Scattering 

Calculating the scattered electron wave, i.e., numerically solving (4a)-(4c), is a 
multi-scale problem. To see this, we compare the wavelength A of the electron to the 
(1) size of the region of interest, (2) the variation of the electrostatic potential, and 
(3) size of pixels where the intensity is formed. TEM imaging typically operates 
with energies between 200 and 300 kV resulting in electrons with a wavelength 
between 0.0025 and 0.0020 nm. In the example below, we consider the case of 
200 kV TEM imaging, so A ~ 0.0025 nm. 


Size of region: When imaging biological specimens, a typical region of interest 
29 C 2 is a rectangular box where the smallest side is the slab thickness, 
which typically is >50 nm. For simplicity, let (29 be a cube with a side length 
of 50 nm + 20,0004. A straightforward numerical discretization of (4a) would 
require about ten points per wavelength (in theory, two points per wavelength 
is enough but with limited precision arithmetic, one needs 10-20 points per 
wavelength to reliably solve Helmholtz type of wave equations using a finite 
element method [121]). In this case we would have to sample (10- 25,000)? ~ 
1.6-10!° points. Even in single precision (four bytes per value), this would 
require 58,200 Tbytes of memory! 

Contrast: To compare the variation of the electrostatic potential U against the 
wavelength A, we introduce 


C= Pe ena max 
27h? xEN 


u(x)|. 


For water U(x) + 4.9V so C  1.26-107, for proteins 30 V < U(x) < 
70V, so C & 1.8-10~*, and for single colloidal gold particles (typically 
embedded in biological specimens as fiducial markers) U(x) ~ 280 V so 
C 72-10%. 

Size of pixels: If we image at 65,000x magnification (a rather common setting for 
imaging subcellular structures in biology), the effective pixels size of a typical 
TEM detector becomes 0.23 nm ~ 115 A. Hence, the intensity (square norm 
of the scattered wave that is a solution to (4a)-(4c)) will be averaged over 2D 
squares with a side length of 115 A. 


In summary, the multi-scale nature of numerically solving (4a)—(4c) requires one to 
consider various approximations. 
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Geometrical Optics Approximation 
Geometrical optics is an approximate treatment of wave propagation where the 
wavelength is considered to be infinitesimally small (semi-classical approximation). 
The idea is to represent the highly oscillating solution as a product of a slowly 
varying amplitude function and an exponential function of a slowly varying phase 
multiplied by a large parameter. 

The starting point is the Wentzel—Kramers—Brillouin (WKB) approximation 
for the stationary Schrédinger equation (4a). For a first order approximation, this 
method seeks an asymptotic solution of the form 


W(x) = a(x) exp (iS(x)/h) + OA) (8) 


where the amplitude a and the phase S' are smooth real-valued functions indepen- 
dent of #. Substituting (8) into the stationary Schrédinger equation (4a), equating the 
real and imaginary parts, canceling the common phase factor exp(i S(x,t)/h) that 
appears throughout, and ignoring ?-order terms (first order WKB approximation) 
give us 


| V S(x) + eA(x)|? = 2meU(x) + 2ME (9) 
div]a(-)°(V S(-) + eA(-)) |(x) = 0. (10) 


Thus, within the first order WKB approximation, (4a) is solved by (8) in which the 
phase function S solves (9) (an eikonal equation of Hamilton—Jacobi type), and the 
amplitude term a solves (10) (transport equation). 

The eikonal equation (9) can be solved by the ray tracing method [147], 
which in turn is based on the method of characteristics. In brief, one introduces 
a family of curves (rays) s +» y(s) which are perpendicular to the level curves 
(wavefronts) of S. These rays define a new coordinate system where the eikonal 
equation reduces to a far simpler, linear, ordinary differential equation. This ordinary 
differential equation for the phase S can be solved simply by integrating along the 
aforementioned ray, leading to 


S(y(t)) = S(y@)) + i H(y(s)) — e9(8)- A(p(s)) ds. (11) 


In the above, H: R? — R is given by H(x) := /2meU(x) + 2mE and the ray y 
is given by the Lorentz equation: 


d 
<(# (v9)¥00)) = V A(y(s)) + eB(y(s)) x ws) (12) 


where B = curl A is the magnetic field [79, eq. (57.11)]. Finally, combining (11) 
with (8) and (6) gives us 
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: i f' ——_ 
T° (U)(x) © Wo(x) exo (5 [ H(y(s)) — J2mE as) (13) 
0 
with wo denoting the incident un-scattered electron wave. 


Validity of the WKB Approximation 
In our specific setting, the first order WKB approximation is valid if 


| EE) ge, (14) 
a(x) 2 

For TEM imaging, the above is valid for electron motion in macroscopic fields, 
far from their singularities. The situation for scattering in microscopic fields is less 
obvious. One issue is that the eikonal equation (9) can become singular (caustic 
formation) at points where rays intersect and the amplitude blows up. There could 
also be several ray trajectories joining the two points, even though this particular 
issue can be handled within the semiclassical limit, see, e.g., [89, theorem 1.2]. 


Remark 5. The expression in (13) only makes use of the first order WKB approx- 
imation. The correct semiclassical limit to (4a) contains several phases, so the 
asymptotic solution is a sum of functions of the form (8). 


Remark 6. Gaussian beams is another high frequency asymptotic model that is 
closely related to geometrical optics. Here the phase is complex-valued, so there 
is no breakdown at caustics. The solution is still assumed to be of the form (8), but 
it is concentrated near a single ray of geometrical optics, see [113, 167] for further 
details. See also [142] for a closely related approach. 


The Small Angle, Projection, and Weak Phase Object Approximations 
The following approximations all aim to further simplify (13). These approxima- 
tions apply in particular to TEM imaging of weakly scattering specimens, such as 
unstained biological specimens. 


The Small Angle Approximation 
If leU (x)| <_E, which typically holds for TEM imaging in life sciences, then 


m 


5242 U@). 


1 1 
5H) —J/2mE = 5 V 2meU(x) +2mE — J/2mE x 
Hence, introducing o := me/kh?, (13) becomes 
t 
To (U)(x) © Wo(x) exp (i f U(y(s)) as) forx = y(t) € Mout (15) 
0 


This approximation is usually referred to as the small angle approximation. 
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The Projection Approximation 

Assume §2 is thin enough that we can disregard the curvature of the electron 
trajectories, i.e., the un-scattered electrons travel along straight lines parallel to the 
direction @ of the incident plane wave Wo. Assuming x + tw € Iout, the above 
considerations combined with (15) result in 


To (U)(x +tw) © Wo(x +tw) exp (i> [ U(x +s@) as) forx € In. (16) 
0 


Furthermore, one frequently replaces wo(x + tw) with Wo(x), which is perfectly 
fine when Wo is a plane wave traveling along w (the difference is then merely a 
constant pure phase factor). This is the projection assumption [78]. 


Weak Phase Object Approximation 
The weak phase object approximation is simply based on linearizing the exponential 
in (16), i.e., 


To (U)(x + tw) © Wo(x +10)(1 +io [ U(x + s@) ds) forx €Iy. (17) 
0 


Other Approaches 

A variety of approaches have been developed for simulating electron scattering in a 
TEM. Most of them, like the Bloch wave method, are only of interest for modeling 
TEM imaging in diffraction mode which is important in electron crystallography 
and in some material sciences applications. There are however approaches that 
are also applicable for computational treatment of conventional bright-field TEM 
imaging of amorphous specimens. 


First Order Born Approximation 

The starting point is to reformulate (4a) as an integral equation (Lippmann— 
Schwinger equation). The latter can be solved by means of an iterative procedure, 
and the first step in that procedure yields the first order Born approximation. This 
results in an affine model for 7,°°, the details are given in [51, section 4.3]. Related 
approximations of 75° are the paraboloid approximation [90] and the thick-phase 
grating approximation [189]. 


Multi-Slice Method 

Here one assumes that (x) = (x) exp(ikx -@) where ¢ is the slowly varying 
part whose sampling in real space may be spaced many wavelengths apart. Inserting 
into (4a) and making use of the Lippmann—Schwinger formulation result in an 
equation for the slowly varying part @. The differential operators in this equation 
can now be separated into operators along w and operators acting in w+. Ignoring 
back-scattering yields a recursive scheme for calculating 73° on Iu, see [146, 
section 3.4]. 


Mathematics of Electron Tomography 957 


Distorted Wave Born Approximation 

The idea here is to express the potential as a sum of two terms, U = Up + AU where 
Up is not necessarily small and AU is sufficiently small that only linear terms in the 
expression for the scattered electron wave need to be retained. This expression for 
U is inserted into the Lippmann—Schwinger formulation of (4a), thereby giving rise 
to a series expansion [146, eq. (7.8)] where one can collect terms in AU and Up. 
Retaining a first order correction for Up results in the approximate description of the 
scattered electron wave, see [146, section 7.2] for details. 


Optics 


After interacting with the specimen, electrons pass through the TEM optics as they 
migrate from the specimen exit plane I to the image plane Ijmag (note that the 
former is not necessarily orthogonal to the optical axis whereas the latter always is). 
An obvious role of the optics is to magnify the image. Another equally important, 
but more subtle, role is related to phase contrast. The optics is necessary to make 
phase contrast visible, which is important for HRTEM imaging. Phase contrast 
would be lost if one would measure intensity data directly on Iyut. 


Remark 7. There are situations when one can, at least approximately, reconstruct 
the electron wave function from measured image intensity. Some of these techniques 
involve unconventional imaging modes, of which the most successful is off-axis 
holography using an electron biprism, see [79, section 63.2]. 


Thus, any forward model that seeks to account for phase contrast must model the 
TEM optics. This is especially important for imaging weakly scattering specimens, 
like unstained thin biological specimens. 

Here we only state the model for the TEM optics and refer to [51, section 5] 
for its derivation. See also [80, 161, 162] and [79, sections 64-66] for a thorough 
description of models for various electron optical elements. 


The General Setting 
The optics of the TEM is the portion between the specimen exit plane I and the 
image plane Imag (Fig. 2). Here there is no specimen, but there are strong magnetic 
fields that can deflect the electron beam in a desirable manner. Disregarding 
polarization and the precession imposed by the electron spin (which does not 
appreciably affect the motion of the electron), the TEM optics is fully characterized 
by a specification of apertures and the magnetic vector potential that corresponds to 
the aforementioned magnetic fields. 

We define the optics operator as the mapping that associates an electron wave 
in Fut to the corresponding wave in Imag. In the general setting, it is the solution 
to (4a) with U = 0 (no specimen) and appropriate boundary conditions: 


958 O. Oktem 


Fens Vtocal Timag 


y Dp , 


Fig. 2 The optical set-up (§2 and {2p are in reality much smaller) for a single thin lens with an 
aperture in the focal plane (section on p. 960). The case for lens-less imaging (section on p. 960) is 
obtained by removing the lens and its aperture. The incident electron wave Win (plane wave from 
left traveling along the optical axis @) scatters against the atoms in the specimen characterized by 
U: 2 — C (scattering potential) with support in a slab {2 (light-gray region). The subset (29 C 2 
(dark-gray region) is the region of interest. The operator 73°(U) in (6) maps yin on Tin to Wee on 
Tsu. The operator T°? in (19) maps an electron wave on the object plane J%j; onto an electron 
wave on the image plane mag. Finally, Tiltcor defined on paragraph on p.961 maps the electron 
wave on the specimen exit plane J, onto the wave on the object plane [pj 


1 ; 2 2 
Ey = =| —-ihV+eA] y on R’ \ &, 

2m 
w= Wsp on Mout, (18) 
w fulfils (4c). 


Since Imag ‘= @+ + po, the optics operator is given as 
TP (Wp) (x) = W(x + pw) for x € w+ and y solves (18). (19) 


The magnetic vector potential A in (18) is given by the magnetic fields generated by 
the optical system and the electron wave W.p on Iout is the scattered electron wave 
leaving the specimen. 


Ray Optics 

Many electron optical elements, including electron lenses, are modeled by con- 
sidering geometric optics approximation of (18) where the electron is a point-like 
charged mass whose motion is governed by the laws of classical mechanics [162, 
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chapter 1] and [79, p. 1262]. The relation between the wave mechanical and 
classical formulations can in fact be seen directly from (12). This shows that the 
eikonal curve derived within the WKB approximation coincides with the classical 
electron trajectory, so the fundamental laws of geometrical electron optics are 
an approximate consequence of wave mechanics based on the first order WKB 
approximation. In conclusion, electron lenses and their aberrations are adequately 
modeled using geometrical charged-particle optics. Modeling diffraction by an 
aperture (which is an opaque screen with a suitable opening) needs however to 
be based on wave mechanics. 


Remark 8. A theory of electron optics that does not make use of geometric optics 
approximation becomes quite complicated, see, e.g., [79, sections 59.4—59.6] [111]. 
Furthermore, there are many other elements of the TEM that we do not model, 
notably alignment coils, stigmators, and phase plates. Stigmators are used to correct 
for the unavoidable third-order (and possibly higher) spherical aberration of the 
objective lens [80, chapter 41]. 


The Optical Set-Up 

The most important component of the optics is the objective lens, which provides 
the first stage of magnification (about 20-50 times). It is followed by a number of 
projector lenses that provide further magnification. Apertures are present at several 
places, e.g., in the back-focal plane of the objective lens (Fig. 1). 

To model phase contrast imaging in ET, it turns out that one can model the entire 
TEM optical system as a single thin lens with an aperture in its focal plane as 
illustrated in Fig. 2 (see Remark 9 for an motivation). Within this setting, except 
for the specimen exit plane Ij, all the other planes are parallel to each other and 
orthogonal to the optical axis, i.e., orthogonal to w. Since the magnification of the 
single thin lens corresponds to the magnification M of the entire optical system 
(objective and projector lenses taken together), we get 


M=p/q and 1/f =1/p+1/q. (20) 


Here, f is the focal length of the lens, and g, p > 0 are the distances from the lens 
to the objective and images planes J; and Imag, respectively. This set-up does not 
correspond to a physical optical system, so one needs to set values for f, p, and qg 
(section “IIlumination and Optics Parameters’). 


Remark 9. The nontrivial part in replacing the TEM optical system with a the 
system in Fig.2 is to argue that the thin lens in Fig.2 has the same aberration 
properties as the objective lens. The motivation for this is as follows: Due to the 
initial magnification by the objective lens, the angular range of electron beams 
impinging upon the first image plane is very small relative to the angular range 
of electron beams entering the object lens from the specimen. Since aberrations are 
worse at high angles than at low angles, the lens that is most affected by aberrations 
will be the objective lens as it has to deal with largest range of angles in the whole 
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microscope. The lenses below the objective lens (projector lenses) also magnify, and 
as the magnification increases, the range of angles that each subsequent lens must 
deal with is reduced. The first lens of the projector system (sometimes called the 
“intermediate” or “diffraction” lens) matters a little bit, but one can forget about all 
the other lenses. Even though they provide a huge amount of magnification, they 
have virtually no influence on the final image resolution. 


Lens-Less Imaging 

This is the simplest model where the optics is replaced by free space propagation 
corresponding to the distance between the parallel object and image planes. Thus, 
one ignores any magnetic fields (so A = 0, see text preceding (5)), apertures, and 
the effect of specimen tilt (see paragraph on p.961). Despite its simplicity, this 
model is much better than simply assuming the intensity is taken directly at the 
specimen exit plane. The reason is that free-space propagation gives time for the 
phase shift induced by the specimen to develop into visible contrast, thereby partly 
addressing the phase problem. 

Stated mathematically, free-space propagation is given as the solution to (18) 
with A = 0 and Ix replaced by Ij. The resulting equation can be solved 
under the Fresnel approximation and the solution is expressible in terms of a 2D 
convolution in the object plane: 


T°? (Wep)(X) X exp (ik(p _ amen (Wsp(qo a -))(x) forx € ot, (21) 


where, for d > 0, 


k 

Ty? (p(x) := ar {PSE @ p(x) (22a) 
‘ _k 

PSF? (x) := exp (i sll): (22b) 


Single Thin Lens with an Aperture 
Here we consider the setup in Fig. 2. The optics operator can then be modeled by a 
suitable combination of free-space propagation (22a), a model for a thin lens, and a 
model for diffraction by an aperture. 

For simplicity let us disregard the correction for the specimen tilt (section “Model 
Refinements”). After some lengthy manipulation, shown in [51], one arrives at the 
following explicit expression for the optics operator: 


P(x) 


TT? (Wsp)(x) © QnyM 


{ PSF? ® Vplgo + H(=) forx €wt, (23) 


Here, M = (p — f)/f denotes the magnification, ® is a pure phase factor (so 
| O(x)| = 1) whose precise expression is given in [51, eq. (21)], and 
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PSF°P (x) := Foal As(E- + fo) exp (iw(| . P))| (x) forx et. (24) 


The Fourier transform of PSF°?, which by above has a closed form expression, is 
usually called the Contrast Transfer Function (CTF). The function Ay: ot >R 
is the characteristic function for the aperture © C T[yfocqi (pupil function) and 
W: S* xR — R encodes the phase shift associated with imperfections on the optics 
(e.g., defocus, aberrations). The latter has an explicit expression [79, eq. (65.29)], 
which in the context of ET can be significantly simplified since terms accounting 
for astigmatism and higher order aberrations can be ignored [79, eq. (65.30)]: 


1 /C, 
WO) = —Gpi(Gr-2Az) forr eR. (25) 
The constant C, is the third-order spherical aberration of the lens and Az is the 
defocus (the deviation from the focal length and error in the positioning of the 
specimen plane). 


Remark 10. The convolution with the optics PSF in (24) is actually taken against 
v( — q@ — -). The minus sign in the argument of y comes from the fact that a 
single thin lens reverses the image. The actual optics consists of several lenses and 
microscope manufacturers do compensate for such effects, so we may disregard 
these minus signs. 


Model Refinements 


Specimen Tilt 

When the specimen is tilted, i.e., when the slab 2 is not orthogonal to the optical 
axis, then one needs to migrate the electron wave from the specimen exit plane J out 
to the object plane I%,j;. We define Tiiltcor’ as the operator that maps an electron 
wave on the specimen exit plane to the corresponding electron wave on I4,;. Since 
Iyut and Ip; are not parallel, this amounts to a free-space propagation between two 
non-parallel planes. 

Free-space propagation between non-parallel planes has been worked out for 
electromagnetic waves in [6, 125], but the methods are also applicable to TEM 
imaging. Approaches in TEM imaging are based on modifying the point spread 
function for the lens operator (“tilted CTF’), see, e.g., [78, section 3.3] and 
[184, 185]. 


Partially Incoherent Illumination 

For perfect coherent illumination, the incident electron beam should have no energy 
spread (perfect temporal coherence) and electrons are emitted from a vanishingly 
small source (perfect spatial coherence). In practice, energy spread may reach a 
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few eV depending on the type of electron source and the finite source size spreads 
the directions of arrival of the incident electrons at the specimen input plane 
Fin. 

The standard approach for accounting these effects is to represent each of them 
by a damping in the Fourier space of the optics PSF. The details are given [79, 
sections 66.2—66.3], below we simply state the resulting modified optics PSF that 
replaces the one in (24): 


PSF"(x) 2= Fya| Av(Z- + fo) exp (iW ((-F)) Bge(|-P) Ean(I-P) ] 


The functions E,,,; and F,j,¢ are the envelope functions that accounts for the energy 
spread and the extension of the source, respectively: 


E(t) = exp ( = ap cP?) 


4 f4 m~c 
heap CG 2 
Esize(t) = exp ( = g pat (Ft _ Az) ) 
In the above, v,, denotes the mean energy spread, 
1 2 Vace 
oi a a where ¢:= — 


© Uacc(1 + €Uacc) “ame 


with Uscc denoting the acceleration voltage of the source, C,. is the chromatic 
aberration of the objective lens, and a, is the aperture angle of the beam furnished 
by the condenser. 

Detection 

Intensity Generated by a Single Electron 


As the electron wave y reaches the image plane Iimag = w+ + po (Fig. 2), it forms 
an intensity distribution, a process that is encoded by the intensity operator: 


T(W)(x) = |W(x + po)!’ forx € wt. (26) 


If the scattered part is a small perturbation Ay to the incident wave Wo, then it is 
natural to linearize the intensity: 


T(Wo + AW)(x) © |Wo(x + po)|” + Woe + po) AW + po 
+ Wo(x + po) Aw(x + po). (27) 
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The Total Intensity and Its Detector Response 

The micrograph is given by the intensity generated from all electrons reaching 
the image plane. In an ideal detection system, this would model the image in the 
micrograph, but all detection systems introduce noise and distortions which we now 
model. 

First, one can treat each electron independently (independent electron assump- 
tion, see Sect. 4). Next, the models for the distribution of intensity and detector 
response are identical for each electron. Hence, in the absence of stochasticity, the 
micrograph is simply given by the detector response distribution for a single electron 
scaled by a factor corresponding to the number of electrons. 


Detectors Types 

Charged Coupled Device (CCD) cameras, or more precisely detectors based on 
slow-scan CCD sensors, have become the detection technique of choice in ET, so 
here we consider the modeling of such detectors. 

Detection of electrons by a slow-scan CCD sensor can be divided into three 
separate stages [197]: (1) the conversion of incident electrons into photons in the 
scintillator (material that converts high-energy radiation like X-rays/electrons into 
visible light), (2) transport of photons from the scintillator to the CCD array (via 
fiber-optic or lens coupling), and (3) conversion of photons into electrons and 
the readout of the resulting digital signal. Each of these steps can be modeled at 
different levels of accuracy. The model outlined below is accurate enough for most 
computational treatments of micrographs. 


Remark 11. It may seem strange to first convert an incident electron to photons, 
which are then converted back to an electric signal (electrons). The problem with 
using incident electrons directly to generate electron-hole pairs at the CCD is that 
they are too energetic, so they saturate the CCD, see, e.g., [146, p.389]. In this 
context, it is worth mentioning the new generation detectors that directly detect 
the electron without an intermediate conversion to photons, and therefore posses 
significantly better sensitivity and noise properties [55]. 


Detector Response Distribution for a Single Electron 

Consider a slow-scan CCD sensor placed at the image plane Imag, so the scintillator 
entrance surface of the detector is Ijmag. The single electron intensity distribution, 
which is the intensity distribution generated by a single electron at the scintillator 
entrance surface, is now given by 


I(x) :=Z(w)(x) forx €or. (28) 


The corresponding detector response distribution at the plane adjacent to the 
CCD, which models for the detector response per unit area (given as ADU per unit 
area) to a single electron in the absence of stochasticity, can then be modeled as 
[146, p. 391] 
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X +> Coain(x){7 ® PSF“ V(x) + Iy(x) forx €or. (29) 
ol: 


Here, Cgain is the overall gain that measures the average number of digital counts 
that a single incident electron gives rise to, J, is the distribution of the background 
signal, and PSF** is the detector response function that models the spreading of 
the signal generated by a single incident electron within the scintillator and fiber- 
optic/lens coupling. All these components can be determined from specifications 
of the detector and/or estimated from specific calibration measurements (section on 
p. 982). 


Total Detector Response Distribution 

Let No denote the image dose, which is defined as the average number of incident 
electrons per unit area at Ijmag used when the micrograph is acquired. Then, (29) 
combined with the independent electron assumption allows us to model the total 
detector response distribution in the absence of stochasticity as 


C(x) := NoCeain(x){7 ® PSF*'}(x) + Noly(x) forx €w*. (30) 
ot 


Characteristics of the Noise 

To model the detector response for a pixel in the micrograph, we need to integrate 
the total detector response distribution in (30) over the pixel and model the 
stochasticity. 


Shot Noise 

The emission of electrons at the source (section “Illumination’’) is a stochastic 
process. This gives rise to shot noise whose influence is most severe in low-dose 
micrographs, such as in ET when one records multiple micrographs of biological 
cryo-specimens. 

Shot noise is not Gaussian but depends on the signal level. It originates from 
the fact that emission of electrons is a Poisson process, so the actual image dose is a 
Poisson distributed random variable with mean equal to the nominal image dose No. 
To model shot noise, consider first the single electron intensity distribution J in (28). 
Each point x € + gives rise to an intensity distribution at the plane adjacent to the 
CCD: 


y 1 Cgain(x) I(x) PSF@'(x — y) + I(x) for y € wt. 
Now, consider a micrograph acquired using a image dose of No and let C(A) denote 


the corresponding detector response from a (measurable) set A C w+. Due to the 
Poisson stochasticity of the total dose, it is given by 


C(A) = i [. Ceain(x) I(x) PSF" (x —y)+ Iy(x) dN(x) dy, (31) 
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where N is a Poisson random measure (see [34, chapter VI]) with mean given by 
No dx, i.e., 


N(A) ~ Poisson [mf 1x], (32) 
A 


Remark 12. The expression in (31) models the noise in a continuum setting. The 
reason for insisting on a continuum model is that there is no natural pixelization 
of the image plane (scintillator entrance plane), which is where the Poisson 
stochasticity is realized since the electrons generate intensities in that plane. Similar 
models for Poisson distributed data when data sampling is in continuum are given 
in [83, 84]. 


Pixelization and Read-Out Noise 
Besides the shot noise, there is additive and multiplicative noise. Both are related to 
a CCD pixel, so we next consider the pixelization in the CCD. 

A pixelization in the CCD corresponds to a finite tessellation {Aj}; of the 
bounded subregion of w+ that constitutes the CCD detector. The subsets A j Tepre- 
sent pixels and are typically given as square regions centered around some suitably 
chosen point x; (often the midpoint). Disregarding multiplicative stochasticity, 
measured data from a single pixel can be modeled as a sample of the stochastic 
variable 


C%(j) = C(Aj) + EU) for j = 1,...,mae — 


In the above, C is given by (31) and E(j) ~ Normal(j;,0;) is a Gaussian random 
variable with mean jz; and variance o; that models additive read-out noise, and 1 get 
is the total number of pixels. The corresponding stochastic model for a micrograph 
is the stochastic nget-vector 


cia = (C“**(1), re C*"* (nace). (34) 
Probability Distribution of Image Data 
It is quite difficult to derive a closed form expression for the probability distribution 


dat ; f 
of C“"* whose samples represent micrographs. One can however derive closed-form 
expressions for the expected value and covariance: 


j 


Cov[o(j), cH ()] = No ie | I i _ [esin ed Casin(9) 


PSF"(x — y) PSF#(z — ie) dy dzdx + Cov [EW). E())]. (35) 
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The Measured Image Data 
The detector operator T* maps a single electron intensity distribution to the 
expected detector response distribution, so it is defined through the relation 


E[C*(j)] = in TD) (x) dx + fj Ge) 


J 


where J is the aforementioned single electron intensity distribution. From (35) we 
get 


T(1)(x) = NoCgain(x) {I @ PSF } (x) + Nolp(x) forx €o, (37) 


so, T*(1) equals (30), the total detector response distribution in the absence of 
stochasticity. 

A common approximation is to replace the integration over A; in (36) with a 
point-evaluation at a suitable point: 


E[c™ (| ~ IA j{T*") @ PSF" (xj) + Hj. ee) 


Here, |A | is the pixel area, x; € Aj; is some suitable evaluation point (typically 
the mid-point), and PSF" is the associated pixel-shape function. 


Remark 13. Further simplification of (38) is obtained when one considers measured 
data that has undergone basic pre-processing (section “Basic Pre-processing”’). Then 
Ceain is constant and expected background signal jz; can be estimated and deducted 
from data, so we may set 4; = 0. It is also common to consider the simplest pixel- 
shape function PSF?” = 6, ,. With these simplifications in place, the measured data 
associated with pixel j is modeled as 


E ong )| ~ |A; |NoCeain{ 1 ® PSF“t h(x). (39) 


Forward Operator for Combined Phase and Amplitude Contrast 


The forward operator is defined as 7: 2 — #, where 2 is a vector space of 
functions U: §2 — C4 that represents a scattering potential (section on p. 952) and 
H is a vector space of functions x + g(w,x) € Rx for x € w+. Here, the 
function g(@, -) represents the expected detector response distribution at the plane 
adjacent to the CCD (just prior to sampling and read-out) when the TEM image is 
acquired using incident electrons traveling along @ € S?. 

In order to derive an expression for the forward operator, we first observe that 
by (6), (19), and (26), the single electron intensity distribution is given as 
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I(U)(@,x) =LoT?oT(U)(x) x eat. (40) 


The forward operator corresponds to the expected detector response distribution, so 
by (36)-(37) we get 


T(U)@.x) = T#(1(U)(@, +) («) 


re No(@)Cesin{ (UV(o, -) @ PSF \(x) forxe@t. (41) 


Here, No(@) denotes the image dose used to acquire the micrograph, J is the 
corresponding single electron intensity distribution given by (40), and the last 
approximate equality follows from setting J, = 0 (Remark 13). 

To get a more explicit expression for 7, we now consider various approximations 
of the single electron intensity distribution /(U): 


Lens-less imaging: This is the setting considered in section on p. 960 that results in 
the model (21) for 7°?, so (40) becomes 


k2 2 
I(U)(@, x) © CG (42) 


{Psre, @ TSU)qo + -)\cx) 


where PSF? , is given by (22b). 
Single thin lens with an aperture: This is the setting considered in section on p. 960 
that results in the model (23) for 7°, so (40) becomes 


2 


I(U)(@,x) & (43) 


{ PSF” @ T(U)(qo + -)\(=) 


(22)* M2 
where PSF°? is given by (24). 


We conclude by providing the most common closed form expressions for 7 relevant 
for combined phase and amplitude contrast modeling in HRTEM imaging. These are 
based on combining specific models of the scattered wave 7;°(U) with the above 
expressions of the single electron intensity distribution. 


Standard Phase Contrast Model 
For TEM imaging of thin weakly scattering specimens, such as cryo-fixated 
biological specimens, one can combine a series of approximations that ultimately 
result in an affine forward model. Currently this is the best trade-off between 
computational feasibility and accuracy for ET on HRTEM data. 

The first step is to employ the weak phase object approximation (17), so 


To (U)(x + q@) © Wolx + qo)(1 + ioP(U)(w.x)) forx €@t. (44) 


968 O. Oktem 


Next, assume the real and imaginary parts U;. and Uim of U are coupled like in (7). 
Insert (44) with (7) into (43), linearize the intensity as in (27), and use the fact that 
Wo is a plane wave, so |W%o|? = 1. Then, as worked out in [51, eqs. (37)—(38)], the 
single electron intensity distribution (43) becomes 


HUY.) © 5(1- F25{ PSFRo.-) @ PUnNo.I}(=)). 8 


(20)? 


Here, PSFi3,(@,z) := PSF? +Q(w) PSF°?, with PSF°? and PSF,” denoting the real 


re? 


and imaginary parts of PSF°? in (24), i.e., 
PSF? (x) = F,1 Az(Z- 4 fo) sin | W(|-P)| (x) 
im @ k 


PSF°? (x) = Fal An(E- # fo) cos[1¥(1-P)]] (x). 


Finally, inserting (45) into (41) gives the following expression for the forward 
operator: 


T(U)(@,x) © *eret| {ps ® N(x) 
- aap PSE. ) ® PU. 1G) | (46) 


im 


where PSF (@, x) := | PSE“ @ ® (PSF;) + O(w) PSF? cm) forx €@t. 


ot 


Phase Contrast Model with Lens-Less Imaging 

One can also consider the simpler lens-less imaging model (42) for the intensity. 
Combine the weak phase object approximation (44) with (7), and insert that 
into (42), linearize the intensity as in (27), and use the fact that W is a plane wave, 
so |Wo|? = 1. The end result is an expression of the type (46) but with a different 
PSF. 


Phase Contrast Model with Ideal Detector Response and Optics 
This is the case when Ay = 1, W = 0, and PSF“ = 6,,. in (46). Then, PSF? =0 
and PSF? = PSF" = 6,1, so 


N Cegain 
TU)(o. x) ~ MOMs (1-2 G2 2H)PUee(0 x} (47) 
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Remark 14. If Q(@) = 0, then there is no detectable signal from the specimen. 
This is to be expected since the optics is needed to image phase contrast. 


Forward Operator for Amplitude Contrast Only 


Here we ignore the phase shift imposed by the specimen, so all the contrast in 
the micrograph is assumed to arise from electrons that are blocked from reaching 
the image plane (amplitude contrast). This is adequate for modeling essentially all 
medium-resolution contrast (beyond 2—3 nm) seen in micrographs, e.g., contrast 
from negatively stained specimens. Furthermore, all the early, successful attempts 
at 3D reconstruction relied on such a model [43, 97]. 


The Intensity 

The starting point for modeling amplitude contrast is to consider electrons as 
particles and use a model similar to the Lambert—Beer law for the attenuation of 
X-rays in computed tomography. Thus, consider an incident beam of electrons 
traveling along w € S? and let y:R*> —> Ry denote the mass attenuation 
coefficient of the specimen for an angle w. Hence, 4(x) is proportional to the 
probability that an incident electron traveling along @ scatters with an angle larger 
than a at x. A large value therefore implies that the electrons scatter with high 
angles and a small value means that the specimen is relatively transparent to the 
electron beam. 

After interacting with the specimen, the incident beam gives rise to a (total) 
intensity distribution at the scintillator entrance surface x +» NoI(x) where No is 
the incoming dose and J is the single electron intensity distribution defined in (28). 
The latter is now modeled as [176, p. 157]: 


I(x) = Ip(x) exp(—P(He)(@,x)) forx €or. (48) 


Here, P is the ray transform in (1) and Jp is the single electron intensity distribution 
one would obtain in the absence of a specimen. 


The Forward Operator 
The forward operator is obtained in the same way as (41) is obtained from (40), i.e., 


T (fy)(@, x) No(@)Cusin{ (Ha (0, -) @ PSF \(x) forxe@t. (49) 


Here, /(jtq)(@, -) is given by the left-hand side of (48), No(@) is the image dose, 
and jg: R? — R, fully characterizes the specimen for scattering with angles up to 
a. Hence, if (49) holds, one can pre-process data so that the corresponding forward 
operator equals the ray transform P in (1). 
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Connection to Quantum Mechanics 
The particle model in (48) can be related to the wave model based on the 
Schr6édinger equation (5). To do this we need to take a closer look at [y. 

First, in the amplitude contrast only model the phase contrast is ignored, so Wee 
and Wo have the same phase. Since I = |¥,c|7, (48) yields 


Wee(x) = Wo(x) exp ( — sf. Lo(x + s@) ds). 


Next, consider the projection approximation (16) for the y,.. Equating the above 
expression for ,, with the one in (16) yields 1g = 20Ujm with o := me/kh?. This 
relates the linear attenuation coefficient jz, to the imaginary part Uj, of the potential 
U in (2). This relation however only makes sense if one removes the explicit model 
for the aperture in the phase contrast model (simply set Ay = 1 in (24)). Then, Uim 
acts as an attenuation term that accounts for both aperture and inelastic scattering. 

Another connection is provided through the (elastic and inelastic) scattering 
cross-section. From [176, eq. (6.1)] we get that 


P(x)Na uf x 


—) forx €R’, 
M(x)” 1) mae 


Ha (x) = 


where oy:.S7 —> Rx represents the local (elastic and inelastic) scattering cross- 
section, 0: R* — Ry, is the density, M:IR*? — R, is the local molecular weight (in 
mol per unit volume), and N4 is Avogadro’s constant (in mol per unit mass). Next, 
by [36, theorem 2.6], we can express the solution to (4a)—-(4c) with A = 0 as 


: x \ exp(ik|x 
w(x) © exp(ikx-w) + oe (eae ee 
|x| |x| 
The term Woo: 5? — C is called the far-field pattern and iWoo is the complex 
scattering amplitude, which in turn is related to the local scattering cross section 
by [78, eq. (6)] 


x 


Ou(xX) = lon. vao(=)I dots. 


In the above, Yy C S? is the region defined by the aperture (i.e., the angular range 
that is blocked by the aperture). 


Summary 
The model for the intensity depends on the type of contrast one seeks to model. 


The model presented in section “Forward Operator for Amplitude Contrast Only” 
for the amplitude contrast is only valid for thin specimens, e.g., unstained biological 
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specimens imaged using 100 keV electrons have to be thinner than 10 nm. If the 
specimen is thicker, this approach is only applicable to model low resolution image 
features. 

At higher resolution, like in HRTEM imaging, one needs to account for the 
phase contrast, leading to the combined phase and amplitude contrast model (section 
“Forward Operator for Combined Phase and Amplitude Contrast”). This modeling 
framework depends highly on the model for the scattering operator 7° defined 
in (6). In section “Electron—Specimen Interaction” we have presented several 
approximations to the scattering operator. The most accurate model is the one 
given by the multi-slice method that accounts for multiple scattering and does 
not assume weakly scattering specimen. The downside is that this results in a 
nonlinear model for image formation that is computationally challenging within 
an inverse problems setting. The first order Born approximation leads to an affine 
model that is more accurate than the geometrical optics approximation. It is valid 
whenever kd AU <« az where d is the size of features one seeks to resolve 
and AU is the contrast (actually it is the ray transform of the contrast) [135]. 
Its implementation is however more complex requiring usage of nonuniform fast 
Fourier transform techniques for efficient evaluation of the scattering operator. 
Finally, for approaches based on geometrical optics approximation (13), only the 
weak phase object approximations (17) result in an affine model. 

Currently there is no computationally feasible theory for modeling 2D image 
features in the micrographs at an intermediate resolution range (<1 nm) in an 
amorphous specimen that is not ultra thin (thickness between 15 and 20 nm) [176, 
p. 157]. Models for simulation of such micrographs, see, e.g., [176, section 6.7], 
are computationally to demanding to be used within an 3D image reconstruction 
scheme. This is also confirmed by a recent study in [189] that considers cryo- 
electron microscopy (section on p. 942) of amorphous biological specimens. There, 
one concludes that the projection and weak-phase object approximations are 
satisfied for simulation of TEM image features not smaller than 0.5 nm, so higher 
resolution simulations require more advanced models for the scattering operator, see 
also [78]. In ET life-science applications one typically does not attempt to interpret 
3D image features that are smaller than 3 nm, so a model for the scattering operator 
that is to be used within a tomographic reconstruction scheme can be based on the 
weak-phase object approximation. 


5 Data Acquisition Geometry 


The reconstruction problem in ET is an example of a tomographic inverse problem. 
Data naturally divides into sub-datasets, each containing data acquired when the 
object has undergone a specific Euclidean transformation (tomographic data). The 
associated data acquisition geometry is a specification of the Euclidean transfor- 
mations that describe how the object moves in between the acquisition of the 
sub-datasets. 
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In ET each sub-dataset corresponds to a micrograph, a 2D TEM image, acquired 
using uniform illumination mode (Sect. 2). A tilt-series refers to the series of 
micrographs, and the zero-tilt micrograph is the one recorded when the specimen 
input plane is as orthogonal as possible to the optical axis. The specimen is only 
rotated (no translations) in-between sub-datasets, so the data acquisition geometry 
follows a parallel beam geometry, i.e., all electrons that generate an image travel 
along the same direction and the detector is planar and orthogonal to this incident 
direction of propagation. 


Parallel Beam Geometries 


For parallel beam geometry, continuum tomographic data is represented as a real- 
valued function defined on the tangent bundle T(S7): 


T(S*) := {(w,x) € S?x R?: x € w+}. 


T(S*) provides natural coordinates for tomographic data since (w,x) € T(S*) 
represents data at x in the detector plane when the orientation of the detector is 
given by w € S? (normal to the detector plane). 

In ET data is given on a 3D subset of T(S7), typically be restricting @ € S* toa 
curve Sy C S?. Hence, the subsets of T(S*) we consider are of the following form: 


Definition 1. A parallel beam line complex (also called Orlov’s pencil) is a subset 
Ms, © T(S?) where 


Ms, := {(@, x) € T(S”) : @ € So} for fixed So C S?. 


So is sometimes called the angular data collection aperture [42]. Next, for a given 
region of interest 25 C R°, we define 


Ms5,(Qo) := {(@,. x) € Mg, x = y —(y-@)@ for some y € 2o.}. 


For parallel beam geometries, continuum tomographic data is represented by a 
function g: Ms, — R and the data acquisition geometry is a specification of the 
curve So. 


Examples Relevant for ET 


Here we give some examples of parallel beam geometries that occur in ET. The 
corresponding curves So are shown in Fig.3. For single- and double-axis tilting, 
coordinates (x, y,z) in R* are chosen so that the x-axis is parallel to the tilt-axis 
and the z-axis parallel to the electron beam when the specimen is in the zero-tilt 
position. 
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Fig. 3 Illustrating So C S? for the three most common data collection geometries in ET, single- 
axis titling (/eft), double-axis tilting (middle), and conical-axis tilting (right). The z-axis is the 
electron beam direction, and in the case of single-tilting, the specimen is tilted around the x-axis 


Single-axis tilting: Here So := {(0, sin@,cos@) : —O@max < 0 < Goat where a 
typical value of the tilt-angle is @max = 60°. 

Double-axis tilting: So is here the union of two single-axis tilting geometries, one 
along the x- and the other along the y-axis. As for single-axis tilting, a typical 
value for the tilt-angle is @nax = 60°. 

Conical-axis tilting: Fix 6, typically 6 = 45°, and define 


So := {(sin B sin 0, sin B cos 6, cos 0) : 0 < @ < 180°}. 


In the ET community, the curve So is often parametrized in terms of the Euler angles 
[143, p.6] instead of describing it, like above, as a curve on S$ 2. This is also the 
convention used by most software packages for ET. 
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6 The Reconstruction Problem in ET 
Mathematical Formulation 


We will state two variants of the 3D reconstruction problem in ET, one for contin- 
uum noise-free tomographic data and one for finitely sampled noisy tomographic 
data. 


The Specimen 

In the combined phase and amplitude contrast model (section “Forward Operator for 
Combined Phase and Amplitude Contrast’), the specimen is fully characterized by 
its 3D scattering potential Ujue: R? > C4. In the amplitude contrast model (section 
“Forward Operator for Amplitude Contrast Only”), it is fully characterized by its 3D 
mass attenuation coefficient j44:R* — R,. Since the mass attenuation coefficient 
can be identified with the imaginary part of the scattering potential (paragraph 
on p. 970), we will henceforth use the same notation, U;,,., for both these functions 
and refer to them as the signal. 

Next, the specimen is contained in an infinite slab 2 C R3 given as an 
unbounded region enclosed between two parallel hyperplanes, Ij, (specimen input 
plane) and Ij, (specimen exit plane). We also introduce the region of interest 
929 C &, which is a bounded domain where we seek to recover the true signal 
Utrue. Note that Uue has compact support in £2, but its support is not necessarily 
contained in {29. In ET this is not the case (section “Incomplete Data, Uniqueness 
and Stability’). Finally, we introduce 2 (reconstruction space), the vector space of 
feasible signals. Elements in 2 are complex-valued functions with compact support 
in 92 with reasonable regularity (section on p. 952). 


Continuum Tomographic Data 
Here tomographic data is represented by a function g: Ms5,(29) — R+ where 
Ms, (80) C T(S*) given as in Definition 1. The data space # is a suitable set of 
such functions, typically one requires that it is contained in 7 (M So (Qo), R). 

The inverse problem in ET is to recover the signal Ujue € 2% on 29 from 
continuum tomographic data g: M5, (§20) — R+ where 


g(@,x) = TUme)(@,x) for (w,x) € Ms,(2o). (50) 


Here 7: 2 — # is the forward operator associated with TEM imaging (section 
“Forward Operator for Combined Phase and Amplitude Contrast” for combined 
phase and amplitude contrast and section “Forward Operator for Amplitude Contrast 
Only” for amplitude contrast only). Finally, ET is local tomography problem, 
meaning that the support of Ujue is not contained in £29. 
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Sampled Tomographic Data 

Finitely sampled tomographic data directly corresponds to a tilt-series. To each 
micrograph in the tilt-series, we associate a directional vector w; € So an image 
dose No(@;). The former is given by the angle between the incident beam and the 
specimen input plane Jj, (zero-tilt is when w; _ Jj,), and the latter is the total 
dose used to acquire the micrograph. Furthermore, there is a pixelization {A ;}; 
associated with the CCD chip. If there are m, micrographs in the tilt-series and the 
CCD chip has get pixels, then tomographic data is represented as a vector in IR" 
with m = muyNaet- 

Next, we introduce the sampled forward operator T: 2 — R” as 


TU), j) = E[c*@, j)| fori =1,...,mmandj =1,...,ma (1) 


where C“*"(, 7) is the stochastic model in (33) for the measured value at the 
j:th pixel in the i:th micrograph (the dependency on U is suppressed in the 
notation). From (36) we can express the sampled forward operator in terms of the 
forward operator 7: 2 — # (section “Forward Operator for Combined Phase 
and Amplitude Contrast” for combined phase and amplitude contrast and section 
“Forward Operator for Amplitude Contrast Only” for amplitude contrast only): 


TUM D =f TWyeoi.) dx ~|Aj|{TUV@, +) @ PSEM } (x) + 1) 


x |Aj|T(U)(@,x;). 
(52) 
The last two approximations follow from (38) and (39), respectively. 
The corresponding inverse problem is to recover the signal Ute € 2% on 20 
from a tilt-series g € IR"! where 


g= T (Useue) ae gue. (53) 


In the above, 7 is the sampled forward operator and g"°**(i, 7) (noise component 
of data) is a sample of the random variable C“"(i, 7) — T(Utue)(i, j)- 


Fully Discrete Setting 

This is the case when the reconstruction space 2 is discretized by mapping an 
element in .2 to a vector in R".. The inverse problem is to recover the n-vector 
Ute € R" from sampled tomographic data g € RY where 


g= T U true) 4 ge (54) 


In the above, T: R” — R” is the fully discretized forward operator that corresponds 


to T. 


976 O. Oktem 


Notion of Solution 


An issue that often arise when dealing with inverse problems with noisy data 
and/or uncertain forward operator, like in ET, is that data is not in the range of the 
forward operator (inconsistent data). For such cases, (53) (or its fully discretized 
version (54)) will not have a solution (non-existence). A common approach to 
address this issue is to introduce a relaxed notion of solution by considering 


argmin D(TW), g) for given D:R” xR” > Rx. (55) 
VEX 


In the above, D is the data discrepancy that quantifies the goodness-of-fit. Choosing 
it as the 2-norm in R” yields least-squares solutions, and more generally, deriving 
it from the data likelihood yields Maximum-Likelihood (ML) solutions. Existence 
for (53) with the above notion of a solution translates into existence of solutions 
to (55), see [196] for an overview of such results. In general, such results are highly 
dependent on the (i) space 2, (ii) data discrepancy D, and (iii) forward operator 
T:% >R". 

Besides existence, one also has the issue of uniqueness. As with most inverse 
problems involving function spaces and finite data, if the inverse problem (53) has 
a solution, it will not be unique. Furthermore, in ET the number of data points 
is almost always less than the number of 3D voxels, so the same also holds for 
the fully discretized version (54). Hence, assuming (55) has solutions, there are 
infinitely many of them (non-uniqueness). Handling the issue of non-uniqueness is 
the art of regularization, where the basic idea is to make use of a priori knowledge 
about Utne € 2. 


Expressions for the Data Discrepancy 

Choosing D as a suitable affine transformation of the negative log-likelihood of the 
random variable in (51) (that models measured data g) allows one to interpret (55) 
as Maximum-Likelihood (ML) estimation of (53). 

Retaining the above interpretation of (55), data with additive Gaussian noise 
(with zero mean and known covariance matrix 2) leads to choosing D as the 
Mahalanobis distance. It weights the usual 2-norm with the covariance matrix ¥ 
of data to make the distance metric scale-invariant: 


D(TW).g) = 3(TW) —g)'-E-!.(TU)—g). 


On the other hand, for Poisson data, D is given by the Kullback—Leibler divergence 
in the data space: 


Mathematics of Electron Tomography 977 


m 


D(TW).8) = D> (gi log TU); - TU). 


i=1 


Now, the ET noise model in (34) is more complex. Deconvolving the 
detector noise leads to data that is Poisson with additive Gaussian noise. It 
turns out that the corresponding log-likelihood has a closed form expression 
[16]. If one insists on a closed form expression for the data discrepancy D, 
this is probably the most accurate noise model for images acquired using 
a low photon/electron count and high detector noise. A difficulty however 
is that D becomes non-convex and non-smooth. Therefore, a variety of 
approximations have been suggested, like [15, 32] and [67, 110], the latter 
two based on formulating a Mahalanobis distance that approximates the log- 
likelihood. 


7 Specific Difficulties in Addressing the Inverse Problem 


The inverse problem in ET is associated with a number of difficulties that make it a 
challenging problem. 


The Dose Problem 


This is the single most important factor limiting the usefulness of ET in life 
sciences. As explained below, the dose problem arises due to damage induced by 
the electron—specimen interaction and limits the total number of micrographs that 
can be recorded. 

When a specimen is irradiated by an electron beam, it gets progressively 
damaged due to ionization. The actual mechanisms underlying specimen damage 
are more complex and outside the scope of this review, the reader may consult 
[9,46] and the references therein for further information. For our purposes, the 
implication of specimen damage is that the number of electrons used to irradiate the 
specimen need to be low enough to preserve the structural integrity of the specimen. 
For biological materials, the total dose varies between 1,000 and 10,000 e~ /nm? 
[148, section 3.2], depending on the size of image features that are of interest, 
the type of specimen, and the sample preparation method. Similar thresholds are 
presented by [103, section 3.2] and [19]. Hence, we are roughly dealing with a total 
of 300-3,000 e~ /pixel (at 25,000 magnification using a detector with a pixel-size 
of 14 m) distributed over 60 or 120 images, so micrographs have very low signal- 
to-noise ratio with significant influence of shot noise, see also Fig. 4. 
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Low-dose (about 3000 e~ /nm?) mi- High-dose micrograph taken at 
crograph recorded at 1 um under-focus. 12 1m under-focus at the end of the ET 
data collection. 
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Fig. 4 Micrographs of an in vitro specimen containing microsomes (small spherical shells with 
diameter ~ 60 nm) recorded using a 200kV TEM. Both micrographs represent the same 
rectangular subregion (HX W: 150x106 nm) of a zero-tilt micrograph. Note that the microsome, 
which is visible in the high-dose micrograph (b), is not visible in the low-dose micrograph (a). 
The high-dose micrograph is not used for reconstruction due to specimen damage. (a) Low-dose 
(about 3,000 e~ /nm?) micrograph recorded at 1 {2m under-focus. (b) High-dose micrograph taken 
at 12 tm under-focus at the end of the ET data collection 


As a final note, the dose problem is less of an issue for ET applied to material 
sciences since those specimens are not as sensitive to damage induced by the 
electron—specimen interaction as biological specimens. 


Incomplete Data, Uniqueness, and Stability 


Mathematical uniqueness and stability results are only meaningful for inverse 
problems with continuum data (50) or in the fully discrete setting (54). For both 
cases, results are highly dependent on the forward operator, the data collection 
geometry, and the choice of the reconstruction and data spaces. Here we focus 
on (50), the case with continuum data. 

It is easy to see that the inverse problem (50) cannot have a unique solution. One 
source for non-uniqueness is the phase problem (section “Phase Retrieval’’), another 
is the fact that tomographic data is local, e.g., the support of Use is not contained in 
§2o whereas data is given only on M 5, (S20). Furthermore, even if one can address 
the issue of non-uniqueness, there are strong indications that the inverse problem is 
severely ill-posed due to limitations in the data acquisition geometry (limited angle 
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data). These claims are supported by first considering the situation in a simplified 
setting, namely the standard phase contrast and the amplitude contrast models. 


Standard Phase Contrast Model 

The forward operator is given by (46), i.e., aray transform followed by convolutions. 
Since the real and imaginary parts of the potential are related as in (7), any 
non-uniqueness issues associated with the phase problem are resolved. Hence, 
uniqueness rests upon the ability to uniquely invert the ray transform on M 5, ({2o), 
and stability refers to the aforementioned inversion and the deconvolution of the 
optics and detector PSFs. 


Uniqueness 

If the support of Uzue is contained in the region of interest (29, then uniqueness 
for inversion of the ray transform follows from [122, theorem 3.144] whenever 
So is infinite (uncountable), no matter how small it is. On the other hand, there 
is non-uniqueness for any finite set of directions, no matter how many [122, 
corollary 3.147]. As already mentioned on p.974, ET is a local tomography 
problem. Hence, without additional prior information about Ujye, the ray transform 
is not uniquely invertible [134, section VI.4]. One option is then to reconstruct 
features that are uniquely recoverable (section on p. 994). 


Stability 

Even when Utye is supported in £29 (so there is uniqueness), we still have the issue 
of stability. Uniqueness is by analytic continuation, a non-constructive method that 
is highly vulnerable to noise in the data. Hence, this approach cannot be used to give 
reliable information about Uj. from data. 

Now, any reasonable form of stability is based on the possibility to obtain 
stability estimates in Sobolev spaces, analogous to those for the complete data case 
So = S*, which in turn requires that Sy C S? fulfills Orlov’s criteria (every great 
circle on S” has a nonzero intersection with So) [42, section 2.3] [134, chapter VI] 
[141, chapter 6]. This does not hold for the parallel beam data collection geometries 
that occur in ET (section “Examples Relevant for ET’), leading to severe ill- 
posedness. The ill-posedness is easy to see in the single-axis tilting scheme, where 
the inversion of the ray transform reduces to inversion of the 2D Radon transform 
in planes orthogonal to the tilt-axis. The limitation on the tilt-angle means that each 
such 2D Radon inversion problem is a limited angle problem, which is known to 
be severely ill-posed [40, 114]. Finally, the deconvolution in ET of the optics and 
detector PSFs are unstable operations, further adding to the ill-posedness of the 
inverse problem (50). 


Amplitude Contrast Model 

The forward operator is (49), and the deconvolution of the detector response PSF** 
does not affect uniqueness. Hence, just as for the standard phase contrast model 
above, the issue of uniqueness/stability reduces to inversion of the ray transform on 


M5, (0). 
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General Inverse Scattering 

Here we consider the case when the intensity in (37) is given by the general 
scattering operator in (6). Since deconvolution of the detector response does not 
impair upon uniqueness, we can without loss of generality assume that PSF“ = 
byt 


Uniqueness 

As already argued for, we cannot expect uniqueness from local data, so we first 
assume Ute is supported in £29. Next, non-uniqueness from phase retrieval can be 
resolved by making two measurements, or assuming the imaginary and real parts 
of the complex valued scatterer are related (section “Phase Retrieval’). Hence, a 
reasonable conjecture is that uniqueness holds for (50) under the same conditions 
as for the standard phase contrast model (section on p.979). More precisely, Usrue 
in (50) can be recovered uniquely whenever the real and imaginary parts of Ute 
are related as in (7) and micrographs are acquired at nonzero defocus (for a single 
thin lens section on p.960) or at fixed distance form the specimen exit plane 
(for lens-less imaging section on p.960). These assumptions are needed for phase 
retrieval (section “Phase Retrieval’), so another formulation is that Ute is uniquely 
determined from scattering data ,.(@, -) = 73°(Utrue) for @ € So whenever So is 
infinite. 

What mathematical results support the above conjecture? For full scattering data, 
there are uniqueness results for determining the far field pattern for (5) from a single 
measurement [35, theorem 10] (see chapter » Inverse Scattering). One can also 
prove uniqueness for the inverse scattering problem with the magnetic Schrodinger 
operator in a slab geometry from a single scattered wave field, but this requires 
full knowledge of the Dirichlet-to-Neumann operator [104]. None of these results 
apply to our setting of tomographic data. The result closest to our setting is perhaps 
the one in [66] where one proves uniqueness for the inverse scattering problem 
when measurements are made on the reflected (backscattered) wave. For phase-less 
scattering data, [88] clearly states that uniqueness cannot hold for the problem of 
recovering Ute from the modulus of the far field pattern for (5) associated with a 
single direction w. Here, §2 is assumed to be a bounded domain, and the scattered 
wave fulfills (4c) and an impedance boundary condition at 0. A similar setting 
is studied in [95,96], but this time data is the modulus of the scattered wave We 
(that solves the direct scattering problem) acquired using wavelengths that vary 
continuously in an interval. A uniqueness result that is closest to our setting is 
the one in [90], which considers (5) with lens-less imaging. It turns out that for 
uniqueness one needs two micrographs for each direction, whereas stability requires 
four images. The results are however only proved when Sy C S? is the equator 
and the scattering operator 7,°° is approximated by the first order Born (paragraph 
on p. 956) or the paraboloid approximations. Nevertheless, it is highly probable that 
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the same result also holds when 7;° is not approximated and when Spo an infinite 
set. Furthermore, if the imaginary part of the scatterer is related to the real part, one 
measurement should be enough for formal uniqueness. 


Stability 
All results cited here deal with stability in recovering Utne from full scattering data 
Wsc(@, ‘y= T° Utrue) for @ € Sp. 

There are various stability results for (4a) that assume full knowledge of the 
Dirichlet-to-Neumann map. Results closer to our setting consider the inverse 
scattering problem (5) with boundary conditions (4b)—-(4c), So = S*, and Qa 
bounded domain with smooth boundary. One can, e.g., show that stability increases 
as the wave number k increases [131, Theorem 1.1 and eq. (1.4)]. Similarly, [4] 
establishes Lipschitz stability for determining the low-frequency component of the 
potential. In [140] one considers the case when Utrue = Up + 6U where Up is known 
(typically representing some background) and 6U is the (small) perturbation that is 
to be recovered. The result is that the perturbation can be stably reconstructed up to 
a certain frequency. 

Stability in the context of phase contrast tomography is studied in [90], which 
considers (5) with lens-less imaging and Sy C S? being the equator. They observe 
that numerical stability requires four measurements for each direction. A similar 
result is given in [130], which studies the same inverse scattering problem. If the 
real and imaginary parts of Ute are proportional (a special case of (7)) and two 
micrographs per direction are acquired, then the inversion is stable with respect 
to high-frequency noise. The arguments are however based on the projection 
assumption, but the result is interesting as it differs from situation in conventional 
tomography, which is known to be mildly ill-posed. This observation that phase 
contrast tomography is more stable than conventional tomography (assuming one 
can handle the phase problem) might also extend to the setting when Sy C S? 
does not fulfill Orlov’s criteria (section on p. 979). Hence, we expect wave imaging 
problems to be less influenced by limited angle data than ray imaging problems. 
The intuitive reason is that waves can “probe objects around corners” whereas rays 
cannot, and Fig. 5 illustrates this. There are however no formal mathematical results 
supporting this observation. 


Nuisance Parameters 


The formal statement of the inverse problem in ET in section “Mathematical 
Formulation” leaves out an important difficulty, namely the presence of nuisance 
parameters. These are parameters whose values need to be recovered alongside the 
potential U. We next list some of these parameters and indicate how they can be 
determined. 
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The phantom. Inversion from com- Inversion from com- 
plete ray transform data. plete diffraction tomog- 
raphy data. 


Inversion from lim- Inversion from lim- 
ited angle ray transform ited angle diffraction to- 
data. mography data. 


Fig. 5 Sensitivity of wave and ray tomography in 2D against limited angle data (angular range is 
+60°). Wave tomography is when data is restrictions of y(, -) for @ € So to a line orthogonal 
to w and where y(@, - ) solves (5) with (4b)-(4c). The wavelength of the incident wave equals the 
diameter of small white disc in the phantom in (a). It is clear that the limited angle artifacts 
(bottom row) are much more severe for ray tomography than for wave tomography. (a) The 
phantom. (b) Inversion from complete ray transform data. (c) Inversion from complete diffraction 
tomography data. (d) Inversion from limited angle ray transform data. (e) Inversion from limited 
angle diffraction tomography data (images courtesy of Frank Natterer) 


Detector Parameters 

The detector model outlined in (29) involves a detector response PSF**, the overall 
gain Cain, and the distribution of the background signal. None of these are nuisance 
parameters in the true sense since they can be determined by separate calibration 
experiments in a way independent of the tilt-series. 

The overall gain Cgain can be calculated from a detector specification and/or 
from off-line data that has undergone basic pre-processing (section “Basic Pre- 
processing”), see, e.g., [146, pp. 380-381]. The detector response PSF" is rotation- 
ally symmetric and positive since the spreading of the signal due to the scintillator 
and fiber-optic/lens coupling is rotationally symmetric. Thus, PSF“ is fully char- 
acterized by the Modulation Transfer Function (MTF), which is the modulus of its 
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Fourier transform: F,, .[PSF**'] (€) = MTF (|&|), Here, MTF: R > R4 measures 
how the signal amplitude is transferred for different spatial frequencies, and for 
slow-scan CCD sensors it is commonly parametrized as [176, p. 279], [146, p. 394]: 


b 


a 
MTF — + +c. 
@) lt+aq 1+ 8q 


(56) 


The above parameters can be estimated from off-line calibration measurements, 
see, e.g., [188], which in turn yields the detector response PSF**. Finally, the 
background signal can also be estimated from similar calibration measurements. 


Illumination and Optics Parameters 

These are parameters associated with the illumination and optics. They are inde- 
pendent of the specimen but they can be unique for a tilt-series. Some, like 
the wavenumber k and magnification M, are considered constant and known to 
sufficient degree of accuracy. On the other hand, the image dose and a number of 
electron—optical parameters have nominal values that must be adjusted, either by an 
analysis of the recorded micrographs and/or by performing additional measurements 
after the ET data collection. 


Image Dose 

The image dose No(@) in (37) plays an important role in the interpretation of data 
since it enters into the forward models in (41) and (49). It gives the number of 
electrons per pixel in the absence of a specimen. 

One way to estimate the image dose is based on assuming a constant electron— 
current density Jo (number of electrons per unit area and unit time). This quantity 
can be calculated by a Faraday cage and converted into the number of electrons 
per second. From the exposure time t(w) we then get the actual dose used for 
the imaging by the relation No(@) = Jot(w). Another option is to set No(@) = 
Cmnax(@)/Cgain Where Cgain is the overall gain (see section on p.982 for how it 
is estimated) and Cyax(@) is the maximum pixel value in the micrograph. This 
will always underestimate the image dose. To account for the specimen, consider 
a linear absorption model where the specimen is a continuum medium in the 
slab (2 with thickness h. Hence, if C(@) is the average detector response, then 
C(@) = Co(@) (1 —h(w)) in which Co(@) is the image dose, jz is characterized by 
the specimen, and h(@) > O is the thickness along @ (section “Forward Operator for 
Amplitude Contrast Only”). If the image dose is the same, say Co, for two images, 
then it is easy to see that it is given by Cp = (No(@) — No(@’))/(2—2h(w)/h(@’)). 
A more general approach is to use least-squares to fit a quadratic function to points 
on @ > Chax(@). This uses all data in the tilt-series and also allows one to estimate 
the shape of the specimen boundaries (a slab is the simplest shape but a “meniscus” 
type of membrane is more natural). 
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Remark 15. In experimental protocols, one typically reports the total dose at the 
object plane I; (Fig. 2). It roughly corresponds to the number of electrons per 
unit area at the specimen input plane [;,. This is an important number as it directly 
relates to the dose sensitivity of the specimen. The corresponding image dose is 
obtained by first dividing with the number of micrographs in the tilt-series, and then 
dividing by the magnification. 


Optics Parameters 
These parameters are only relevant for reconstruction methods based on forward 
models that account for the TEM optics, like the standard phase contrast model 
(section on p. 967). 

First, we have parameters that describe the set-up for the optics (section “The 
Optical Set-Up”) illustrated in Fig. 2. This set-up does not correspond to a physical 
optical system. The parameters f, p, and q can be selected freely as long as (20) 
holds. It is however wise to use the convention that f in (20) equals the focal length 
of the actual objective lens. The two relations will then give the values for p and 
q. An advantage of this convention is that the size of the aperture in the focal plane 
Toca) Matches the size of the physical objective aperture. This convention provides 
the parameter d := p—q associated with lens-free imaging, the simplest model for 
the optics (section on p. 960). 

Next, a more advanced TEM optics model, like the one in section on p. 960, 
requires values for electron—optical parameters (CTF estimation). These enter 
into the definition of PSF°? in (24) with defocus being the most important one. 
In ET applications that do not require interpretation of 3D features smaller 
than 4 nm, there is no need for CTF estimation since nominal values of the 
electron—optical parameters are good enough. This is however not the case for high- 
magnification/resolution applications [57]. In general, CTF estimation is based on 
identifying shape and position of the so-called Thon rings in the power spectrum 
density. These are resolution-dependent amplitude modulations caused by the optics 
PSF [79, section 77.2]. This identification is an ill-posed problem [175] and a 
variety of estimation techniques have been developed, mainly for single particle 
applications, see [60, sections 2.37—2.38] and [51, section 8.5] for a more detailed 
overview. Unfortunately, most of these approaches are not applicable to ET since 
micrographs have too low contrast and/or are too noisy. One approach is based on 
strip-based periodogram averaging, extended throughout the tilt-series to overcome 
the low contrast conditions found in ET [57, 191]. Another alternative, that better 
respects the true Poisson characteristics of the noise, is outlined in [186]. 


Remark 16. For material sciences applications, CTF estimation is important due to 
the high resolution/magnification. The estimation problem is in part simpler since 
micrographs have better contrast and signal-to-noise ratio, but more CTF parameters 
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have to be estimated as well. The reader may consult [176, chapter 10] for a survey 
of approaches. 


Specimen-Dependent Parameters 

In ET there are also parameters that depend on the specimen. One such parameter 
relates to local data (p. 974). Since the support of U is not contained in 29 C 92 C 
IR, pixels in all micrographs (besides the zero-tilt image) will have contribution 
from outside (29 (local tomography artifacts), see, e.g., [123, section 4]. These 
artifacts become more pronounced when there is strongly scattering material 
outside (29, as for biological in situ specimens and material sciences. A typical 
approach is to “pad” the data, which here means removing contributions from 
§29 (long-object compensation). The simplest padding is based on assuming U 
equals some constant average value outside (29 (constant continuation), which 
then becomes a specimen-dependent parameter. One can also consider smooth 
continuation in order to avoid introducing edge effects, but that results in introducing 
further specimen-dependent parameters. Yet another approach is to consider a low- 
resolution reconstruction from a larger region of interest, and use that to pad 
data. 

Another parameter is the scaling factor Q associated with the assumption 
in (7) that relates the real and imaginary part of the potential. Current methods 
for estimating the amplitude contrast ratio are based on performing additional 
measurements, such as taking a defocus series. This is unfortunately not possible 
for most biological specimens. 


8 Data Pre-processing 


This section describes data processing steps that are commonly applied to micro- 
graphs in a tilt-series before they are used within a reconstruction method. 


Basic Pre-processing 


The degree of pre-processing of micrographs depends on how much one seeks to 
include in the forward operator. For all ET applications it is highly recommended 
to work with data that has been flat-fielded. Flat-fielding refers to the process of 
compensating for different gains and dark currents in a detector, so a uniform signal 
corresponds to a uniform output. This process also includes gain normalization that 
removes image features associated with the structure of the scintillator crystal and 
fiber-couplings (section on p. 963), see an example of this effect in [146, p. 393]. In 
addition, it is customary to remove high frequency peaks (X-ray peaks) based on a 
local mean value calculations. 
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Alignment 


Acquiring a tilt-series corresponds to sampling a real valued function defined on 
Ms5,(&o) (Definition 1). The nominal specification for this sampling is unfortu- 
nately not accurate enough since the specimen undergoes unintentional movements 
during in between the acquisition of the micrographs. These movements result in 
shifts, rotations, and other types of image distortions. 

Alignment is the procedure that corrects for these distortions and determines 
how M5, (S20) is sampled. Most alignment procedures follow a common four-step 
scheme: 


Initial processing: In addition to basic pre-processing (section “Basic Pre- 
processing”), micrographs in tilt-series often also undergo further processing 
that involves denoinsing and feature extraction specifically tailored for 
alignment. These micrographs are only used during the alignment; to avoid 
losing information, they are not used for reconstruction. 

Coarse alignment: Despite high-precision mechanics, raw tilt series might contain 
large shifts and rotations. Cross-correlation is used to coarsely align adjacent 
images. After coarse alignment, we can assume a smooth trajectory of features 
across images, facilitating subsequent alignment steps. 

Feature tracking: The most common technique for feature detection is to define a 
patch around the feature as a template, and cross-correlate this template with 
other micrographs to search for the feature of interest. Due to high noise in 
images, one often have many false positive detections. 

Determine the geometric relation: Once the 2D location of each feature is estab- 
lished and tracked across multiple images, epipolar geometry can be used to 
establish the geometric relation between the various images. 


The choice of alignment method is based on the type of feature one seeks to 
track. Cryo-fixated specimens, especially in vitro specimens, lack visible features. 
Hence, one has to add fiducial markers to align tilt-series of such specimens. These 
are spherical gold beads that appear as high contrast point-like features in the 
micrographs. Techniques from epipolar geometry can then be used to elucidate the 
geometric relation between the micrographs, see [3] and the references therein for 
more details. 


Deconvolving Detector Response 


The Fourier transform of the detector response is strictly positive (section on 
p. 982), so it is fairly straightforward to deconvolve it, e.g., using a Wiener filter 
or the Richardson—Lucy algorithm [197]. A benefit of deconvolving the detector 
response, instead of including it into the forward model, is that the noise model for 
(deconvolved) data (section on p. 964) becomes much simpler. 
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Deconvolving Optics PSF 


Deconvolving the optics response is only relevant for tilt-series from HRTEM 
imaging where phase contrast is important. 

For lens-free imaging (section on p. 960), this corresponds to deconvolving the 
effect of free-space propagation. Several approaches exist for this within the X-ray 
phase contrast imaging community, see [22] for a survey. 

Optics models that are based on the single thin lens with an aperture model 
(section on p.960) are more involved to deconvolve since the CTF (the Fourier 
transform of the PSF) in (24) has multiple zeros. As an example, for the standard 
phase contrast model (section “Standard Phase Contrast Model’), one has to 
deconvolve PSF,, given in (45). If Q(w) = 0, then PSF,3,(0) = PSF? (0) = 0, 
i.e., low-frequency image information is lost. Hence, the optics deconvolution is 
more challenging for HRTEM imaging specimens that act as “pure phase” objects. 
In general, the challenge in regularizing the optics deconvolution is the difficulty in 
specifying reasonable a priori information about the “true” signal, which here would 
be the micrograph one would obtain without perfect optics. 

The optics is on the other hand needed to make phase contrast visible, so its 
deconvolution is only sensible if it is coupled with phase retrieval (section “Phase 
Retrieval’). In conclusion, it is highly dubious to first deconvolve the optics and then 
try 3D reconstruction if one does not couple that with some kind of phase retrieval. 
Thus, one either ignores the optics or include it into the forward model, so optics 
deconvolution is performed simultaneously with 3D reconstruction. 


Phase Retrieval 


For each direction w € So, we have a phase retrieval problem (also called exit-wave 
reconstruction problem) which amounts to inverting the intensity operator in (26). 
In general this problem does not have a unique solution. 

A common approach is to record two, or more, micrographs of the specimen 
using different imaging conditions (like different defocus values). From such data 
one can then reconstruct the phase of the wave [58]. This principle is put into action 
in electron holography and approaches based on the transport-of-intensity equation 
[99]. None of these approaches are however applicable to ET in life sciences since 
they require multiple micrographs for each direction @ € So, which is often not 
possible due to the specimen dose sensitivity (section “The Dose Problem’). Thus, 
we are left with considering “in-line” approaches that use only one micrograph at 
each direction. 

Let us first consider the simplest model for the optics (lens-less imaging, section 
on p. 960), i.e., free-space propagation. In this setting, phase retrieval within ET is 
identical to the corresponding problem in in-line X-ray phase contrast tomography. 
A variety of methods have been developed for the former, all based on relating 
the real and imaginary parts of the complex refractive index as in (7). The idea is 
to make use of the optics operator 7°?, which in this case is given by free-space 
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propagation (21), and perform the phase retrieval and optics deconvolution in one 
step by inverting Z o J°?. The resulting phase retrieval operation is expressible as a 
convolution, see [22] for a nice survey. For ET, there is no obvious relation between 
the electrostatic and the absorption potentials. Nevertheless, it is common to assume 
that the latter is a multiple times the former, as discussed in section on p.949. 
Thus, the aforementioned approaches from in-line X-ray phase contrast tomography 
should be applicable also on ET data as a pre-processing step for phase retrieval. A 
similar approach should also be applicable when the optics model is more complex, 
such as in (23). 

Finally, one also has the option to include the intensity into the forward operator. 
In that context, phase retrieval is performed jointly as part of 3D reconstruction. If 
one seeks an affine forward operator, then one must linearize the intensity as in (26). 


9 Reconstruction Methods 


Before we describe the various reconstruction methods for ET, we state assumptions 
and procedures that are common for all current methods in use. 


Type of Forward Operator 

Unless otherwise stated, all reconstruction methods assume the forward operator is 
given by either the amplitude contrast (section “Forward Operator for Amplitude 
Contrast Only”) or the standard phase contrast (section on p.967) models. For 
data that has undergone basic pre-processing (section “Basic Pre-processing”), both 
these lead to affine forward operators expressible as 


T(U)(@,x) = C(w) — Col PSF(@, ) ® PUr)(o, x) fore wn 


(57) 
The specific expression for the PSF depends on the model but the overall structure 
remains the same for the amplitude and standard phase contrast models. In this 
context, the term C(@) represents data one would record if there would be no 
specimen. It can be, together with Co, estimated from data (section “Nuisance 
Parameters”), in which case the inverse problem in ET can be recast to a setting 
with a linear forward operator. 


Local Data 

In ET we have local data (see p. 974), i.e., the support of Use is not contained in 
$29. Hence, each micrograph contains the contribution of a larger or smaller extent of 
the specimen, so (53) becomes inconsistent. This is not an issue for reconstruction 
methods that are not sensitive to such issues, like ELT (section on p. 994). Other 
reconstruction methods should either pre-process data as to remove contributions 
from outside (29, or adapt the evaluation of the forward operator and its adjoint to 
account for this contribution (section on p. 985). This is referred to as long-object 
compensation and is based on assumptions about Ute outside (29, see [192] for 
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artifacts that arise in ET when such compensation is not performed in iterative 
methods (section “Iterative Methods with Early Stopping”). 


Handling Nuisance Parameters 

A reconstruction method for ET must handle how to assign values to nuisance 
parameters (section “Nuisance Parameters’’), like @ b+ C(@) and Cp in (57). Unless 
otherwise stated, all reconstruction methods in ET assume these are determined 
before reconstruction. 

One can in principle consider the nuisance parameters as part of the signal 
that is to be recovered and attempt at reconstructing these alongside the initial 
signal. This is theoretically possible when using iterative and variational approaches. 
On the other hand, the forward operator might be nonlinear w.r.t. the nuisance 
parameters, the stability of the inversion w.r.t. the nuisance parameters could vary 
considerably when compared to inversion of the original signal, it is unclear which 
data discrepancy to use, etc. Hence, in practice one adopts an intertwined approach 
in which the signal is first reconstructed keeping the nuisance fixed, then the 
nuisance parameters are recovered keeping the signal fixed [165]. 


Type of Mathematical Results 

From a strict mathematical point of view, a regularization involves a reconstruction 
operator that is well-posed (unique and stable solution) when reconstruction 
parameter(s) are chosen accordingly. Next, regularized solutions must converge to 
a ML/least-squares solution as the data error ||g"°**|| — 0. Besides these two 
requirements, one often also studies convergence rates (estimate of the difference 
between the regularized solution and a ML/least-squares solution) and stability 
estimates (bounds to the difference between the regularized solution with noise-free 
data and noisy data). 


Analytic Methods 


Overview 
Analytic methods are applicable to inverse problems where the forward operator 
has an inverse that has an analytic formula, which is a concatenation of “simple” 
mathematical operators (such as finite number of differentiations to any order and 
integrations, whereas analytic continuation or summation of infinite series is not 
simple). The starting point is the reconstruction problem with continuum data given 
in (50). Assuming this problem has a unique solution, one can define the recon- 
struction operator as the inverse of the forward operator (exact reconstruction). 
On the other hand, if there is instability and/or non-uniqueness, like in ET (section 
“Incomplete Data, Uniqueness and Stability”), then an appropriate reconstruction 
operator should recover a feature, such as an approximate identity, of the signal. The 
actual implementation is given by an appropriate discretization of the reconstruction 
operator. 

In summary, analytical methods lead to algorithms in which the signal is directly 
calculated from the measurements in a single step, without resorting to more time- 
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consuming, iterative methods. In tomographic applications, the numerical analysis 
basically employs filtering, backprojection, and summation operations, as well as 
discrete Fourier transforms. 


Backprojection-Based Methods 

The Filtered Back-Projection (FBP) and Weighted Back-Projection (WBP) methods 
are examples of backprojection-based methods that constitute the standard approach 
within the ET community for solving the reconstruction problem, see [143] for a 
recent survey. 


Data Pre-processing 

All backprojection-based methods assume data is (noisy) samples of the ray 
transform, i.e., the forward operator is (57) with C(w) = 0, Co = 1, and 
PSF(@,-) = 46,1. Hence, data should be pre-processed so that it fits this 
assumption, 1.e., nuisance parameters C(@) and Cp need to be estimated and the 
PSF needs to be deconvolved. 


A Priori Information 

The assumption is that noise is high frequency whereas relevant features of Ujue are 
low frequency. Thus, by providing an approximate inverse, these relevant features 
of Ustrue are recovered while noise is suppressed. 


Reconstruction Operator 

The starting point for both FBP and WBP is an equality that relates the ray 
transform to its back-projection. Let So C S? be an infinite measurable set and 
U ¢ .Y(R?,R). A simple modification to the arguments used in proving [136, 
eq. (2.34)] allows us to prove the following identity on M 5, ({2o): 


UxH =P5(h @ PU) forh < 7(Ms,(Q).R) and H := Ps (h). (58) 


Here, P is the ray transform in (1) and P¢, is the (parallel beam) back-projection 
defined as follows: 


Definition 2 (Back-Projection). Let g ¢ 7% (T(S *),R) and So C S? isa 
measurable set. The (parallel beam) back-projection of g is then defined as 


Ps, (g)(x) := g(@,x —(x-@)@)do forx eR’, (59) 


0 


with dw denoting the surface measure on So. 


Remark 17. SF (T(S 2), R) is the Schwarz space for real-valued functions defined 
on T(S 2), see [136, section 2.2] for the formal definition. Ps, maps such functions 
to functions on R*. Furthermore, if the “?-inner product is considered for both the 
domain and range of Ps,, then P;, is the adjoint of P. 
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The reconstruction operator for the FBP method is now defined as the right-hand 
side of the equality in (58): 


Repp(g) := Ps. (h @ g) for g € .7(Ms,,R). (60) 


Hence, by (58) we get that Rppp(g) = Utue * H whenever g = P(Utne) on 
Ms5,(&o). In this context, h: Ms, — R is called the reconstruction kernel and 
H is the corresponding filter. The idea in FBP is now to choose hi such that H ~ 6, 
since this implies that Rrgp(g) ~ Utrue. 

The WBP method is mathematically equivalent to FBP. It is based on taking 
h = 6,1 in (58). This means, of course, that the corresponding filter H will 
not be an approximation of 6, so (60) does not directly yield a reconstruction 
Of Ute. However, if one can derive an expression for the Fourier transform of 
Hw Ps, (6,,.), then one can use this to recover Utrue by dividing the Fourier 
transform of Ps (g) by the Fourier transform of H, and then apply the inverse 
Fourier transform. This leads to the following reconstruction operator: 


Rwep(g) := F! ee 


F [H] 


For FBP, in order to get a useful reconstruction operator one must choose the 
reconstruction kernel such that convolution with Ps, (h) represents extracting 
some feature of U. The idea is to first derive the reconstruction kernel h for 
exact reconstruction from ideal data, i.e., Ps (i) = 6. The regularized variant that 
provides an approximative inverse for noisy data is then given as a band-limited 
variant of the aforementioned reconstruction kernel. Similarly, in WBP we seek 1 
such that it approximately equals the backprojection of a Dirac delta. 


Adaptation to ET 

The adaptation to ET lies in finding an appropriate reconstruction kernel / for the 
FBP method. The corresponding problem of finding H for WBP is discussed in 
[143, 156]. 

For single-axis tilting (section “Examples Relevant for ET”), the entire 3D ray 
transform inversion problem reduces to a sequence of 2D inversion problems that 
can be handled separately. Hence, in that sense, single-axis tilting does not lead 
to a truly 3D inversion problem. Likewise, the current approach for FBP/WBP 
reconstruction from double-axis tilting data is to split such a data set into two single- 
axis data sets, perform 3D reconstruction for each of these, and then combine the 
two 3D reconstructions by some averaging. This might give the impression that 
reconstruction from double-axis tilting is also not a “true 3D problem”. This is 
however not the case.The two 3D reconstructions will have different limited angle 
artifacts, so it is clear that a spatially dependent averaging will perform better than 
just a regular average. One can derive a partition of unity to use for such a spatially 
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dependent averaging by considering the reconstruction kernel for the 3D problem in 
double axis tilting. 

Let us now consider the reconstruction kernel for general parallel beam line 
complexes in which So is a curve. To derive an equation for the reconstruction kernel 
(filter equation), we first consider the following relation [136, theorem 2.17]: 


1 
F [H\(é) = =| F,.{h(@, -)] (&) dw forh € S(Ms,,R), 
IE | Jsong+ 
where H := Ps (h). Exact reconstruction requires H = 6, i.e., Ps, (h) = 1, so 
J. Foul. 116) do = [gh 
Song 


Hence, an obvious choice of reconstruction filter 1, independent of , is 


[EI 


Feslh(@, @ = Fe gy 
C 


where json etm f 5(m-&)dw. (61) 
So 


This is fine as long as So satisfies Orlov’s criteria (section “Incomplete Data, 
Uniqueness, and Stability”), which ensures that | So né | # 0. Unfortunately, in ET 
this is not the case. A natural modification to (61) is to set the Fourier transform of 
h(@, -) to zero at points € where |So nN é+| = 0. With this choice of reconstruction 
filter, the FBP reconstruction operator in (60) provides a least-squares solution with 
minimal 2-norm instead of an exact solution. 

To get a more explicit expression for h, let As, be the set of all € # 0 in R” 
such that s- is not tangent to Sp and intersects it at a finite number of points, with 
N¢ denoting the number of such intersection points. Also, parametrize the curve 
So by s  y(s) where s € J is an interval and dw = ly (s)| ds. From a direct 
modification of the proof of [136, theorem 2.17], we get 


|v (si ds 


f- Wey) w[A(y (si). -)] &) for € € As, 


F [H\é) = any real “TE-¥G0| 


where 5),...,5y, € J are such that  - y(s) = 0. The above immediately gives us a 
necessary condition for h (filter equation) if H = 6 is to hold: 


Ng 
pe or say (si), -)]E) = 1 forall § € Asp. si 


Thus, a natural choice of h is to define it as the inverse Fourier transform of 
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E76] whenever Nz; 4 0, 
Eb 4 20 Nely(s)| (63) 


0 otherwise. 


Note that there are multiple choices for / whenever Ng > 1 for some &. 
Furthermore, the filter equation (62) is a necessary condition for H = 6, so an 
interesting question is to find sufficient conditions. 


Remark 18. There is a great deal of literature in the tomography community for 
constructing filter equations, i.e., designing reconstruction kernels 4 for 3D FBP 
methods such that H = do. Most of these results are not applicable to ET since they 
deal with data collection geometries (cone-beam or helical cone-beam) that arise 
in medical X-ray tomography and therefore are not parallel beam. Some work on 
filter equations in the parallel beam context can be found in early work on positron 
emission tomography [42]. These results are not applicable to ET since they assume 
So C S? is an open set. 


Remark 19. In the 3D electron microscopy literature there are several publications 
devoted to deriving reconstruction kernels when Sp is not a curve, but rather some 
irregular subset of S*. This is motivated from the needs in single particle analysis 
when one seeks to invert the ray transform on a parallel beam line complex in which 
So is an open set and the sampling is irregular/random within So. These results are 
nicely surveyed in [143]. 


Comments and Discussion 

The main advantage of FBP/WBP is that it is fairly easy to implement, so most 
software packages for ET have at least one of these methods implemented as listed 
in [143]. The methods are furthermore efficient enough to allow for large scale 
3D reconstructions in short time. Next, the reconstruction quality is sufficient for 
answering biological questions that only require interpretation of 3D structural 
details larger than 60-80 nm. 

There are several disadvantages that comes with using FBP/WBP. First, one 
assumes a forward model given by the ray transform, so data should be pre- 
processed accordingly to reflect that assumption. Next is to choose a reconstruction 
kernel to yield an approximative inverse, which in FBP is governed by the 
band-limit that becomes the regularization parameter. Techniques from sampling 
theory can now be used to identify efficient sampling schemes, provide qualitative 
understanding of certain artifacts, and provide guidelines for how to band-limit 
given a sampling scheme. This is well understood in the planar setting (i.e., in 
IR?) for complete and moderately noisy fan or parallel beam data [52, 53, 134], 
but the situation is less clear in 3D and/or when data is incomplete. Guidelines, 
like Crowther’s criterion [145, p. 316], that relate the “resolution” to the sampling 
scheme are therefore not applicable to ET even when data is collected by single- 
axis tilting. Finally, FBP/WBP are not really based on a model of the stochasticity 
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of the data. Both methods are essentially a discretization of an inversion formula. 
Therefore, it is difficult (if not impossible) to devise schemes for selecting the filter A 
that takes into account the specific stochasticity of the data, which in the ET setting 
is essentially Poisson distributed. 

A common approach when using FBP/WBP in ET is to post-process recon- 
structions to remove speckle, enhance features, and reduce limited angle streak 
artifacts. The simplest approach for single-axis data is to average over slices. Most 
software packages offer this possibility to reduce the influence of noise. Other 
post-process approaches seek to reduce the well-known streak artifacts, see, e.g., 
[62, 133] in which the latter uses microlocal analysis to explain the appearance of 
the aforementioned artifacts. 


Electron A-Tomography (ELT) 

ET data are local so the corresponding inverse problem does not have a unique 
solution even for continuum noise-free data (section “Incomplete Data, Uniqueness, 
and Stability”). A natural question is to determine what features that are uniquely 
recoverable from such local data and to provide stable means for their recovery. 

As we shall see, one such feature is the “visible singularities” that provide 
location of certain edges of objects in the specimen. The visibility depends on the 
data collection geometry. Furthermore, it turns out that the recovery of the visible 
singularities is only mildly ill-posed (comparable to inversion of the ray transform 
on complete data). ELT is a local reconstruction method that recovers these visible 
singularities that is based on A-tomography. The description given here is based on 
[151,155], see also chapter » Microlocal Analysis in Tomography. 


Data Pre-processing 
Same as for backprojection-based methods (section “Backprojection-Based Meth- 
ods’). 


A Priori Information 
Relevant edge information can be recovered by suppressing high-frequency phe- 
nomena. 


Uniquely Recoverable Signal Features 

The analysis of signal features that are uniquely recoverable from local tomo- 
graphic data relies heavily on microlocal analysis. Microlocal analysis was initially 
introduced to study how singularities propagated in solutions of partial differential 
equations [86, 168], and its application to integral geometry was first given in [76]. 
Somewhat later, microlocal analysis was used in a series of papers for studying the 
admissibility problem for various restricted generalized ray transforms (integrating 
over families of geodesics) [73-75]. Further applications in integral geometry came 
with [20] (support theorems) and [150] (limited data problems for the ray transform 
in R? and R?), see chapter > Microlocal Analysis in Tomography for a more detailed 
description. 
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The central idea in microlocal analysis is that singularities are characterized not 
only by their location, but also by the high frequencies that cause them. To make 
this more precise, one makes use of the well-known Fourier characterization of 
smoothness, i.e., U is smooth if and only if its Fourier transform decays faster than 
any power of 1/|&| as |&| + oo. Localizing this characterization yields a description 
of the singular support (location of the singularity). The formal definition of a 
wavefront set is given by a further localization, microlocalization, that characterizes 
those directions in Fourier space where we lack smoothness (i.e., direction of those 
high frequencies causing the singularities): 


Definition 3 (The Wavefront Set). Let xo € R” and &) € R” \ {0}. We say that U 
is smooth near Xo in the direction & , if there is a smooth cut-off function ¢ near xo 
(i.e. @ has compact support and ¢(xo) 4 0) such that F [U@]is rapidly decaying in 
some open conical neighborhood of &9. The wave front set, WF(U), of U are those 
points (xo, )) where U is not smooth near xo in the direction &. 


A simple example might aid the intuitive understanding. If U has a jump discontinu- 
ity along a smooth hyper-surface J” C R”, then WF(U) consists of (x0, &)) where 
Xo € I’ and &, is normal to I’. 


Remark 20. The above definition skips some mathematical technicalities, e.g., 
€, is actually a cotangent vector and not an element in the same space as 
xX. This distinction becomes important when one defines the wavefront set for 
functions (or distributions) defined on smooth manifolds. Furthermore, one can also 
introduce Sobolev wavefront sets that encodes Sobolev regularity, see [151] and 
chapter » Microlocal Analysis in Tomography for details. 


Next, for inverse problems it is natural to seek to characterize singularities of the 
signal that are detectable in data. In the context of inverting the ray transform, a 
singularity (x, €) € WF(U) is said to be visible from ray transform data g = P(U) 
if it corresponds to a singularity in WF(g). Thus, visible singularity are those that 
leave traces in data, and these can be characterized if data g is given on a parallel 
beam line complex: 


Theorem 1 (Microlocal Regularity Principle). A singularity (x,&) € WF(U) 
is visible from ray transform data g = P(U) on Ms5,(S20) if and only if there 
is a line in Ms,(820) through x to which & is co-normal (with a few exceptions). 
Furthermore, the recovery of the visible singularities is mildly ill-posed in the sense 
that the singularities in the data g are weaker than those of U by 1/2 Sobolev order 
(good enough to allow stable detection in practice). 


An important special case is when U, as in the example following Definition 3, is 
smooth except for a jump discontinuity along a smooth hyper-surface I”. Then, a 
singularity at x € I is visible if and only if there is a line in M5, ({2o) that goes 
through x and is tangent to M5, ({2o) (with a few exceptions). 
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Remark 21. In order to simplify the presentation, we have left out an array of 
technicalities in the formulation of Theorem 1, see, e.g., [151, theorem 6.3] for 
a rigorous formulation in the single-axis tilting case. Note also that this theorem 
is not dealing with a specific reconstruction method. It is more analogous to a 
“uniqueness” result describing what is possible to recover (assuming one has no 
further a priori information) irrespective of the reconstruction method. 


Reconstruction Operator 
We have seen that visible singularities are signal features that are uniquely recov- 
erable from continuum ray transform data on M5,(§29). ELT is a reconstruction 
method that recovers such singularities from ET data. It is an adaptation of A- 
tomography, which was developed independently of microlocal analysis and first 
introduced in [173, 178] as a local tomographic reconstruction method. 

To describe ELT, we start off by considering Sy = S*. Then, [52, eqs. (4-6)] 
yields the following generalization of (58) form > —1: 


A"(U) * H = Ps(h ® PU)) on M5, (20) for H := Ps A~”(h). (64) 


Here, P;, is the backprojection operator given in (59) and A is Calder6n’s operator 
that is defined in terms of the Fourier transforms as F [A(U)](&) := |&|F [U](é) 
for U € .Y(R?,C). Now, we proceed like in the FBP method, i.e., we first derive a 
reconstruction kernel / for exact recovery of A” (U) by selecting h so that H = 6 
in (64). The regularized variant is obtained by band-limiting the aforementioned 
reconstruction kernel, which in turn provides an approximative inverse of A” (U). 
The case m = 1 is of specific interest since A(U)(x) is unaffected by local 
tomography artifacts (section on p. 985) as its calculation by (64) only involves lines 
passing through a small neighborhood of x. 

In ET, So C S? is a curve. For simplicity, consider the case when So is the 
equator in the (x, y)-plane. Then Ps PU = k * U where 


1 
|x’| 


k(x) := c8(z) with x’ = (x, y). 


Let A’ and A’ denote the 2D variants of A and the Laplacian restricted to the (x, y)- 
plane, ie., A’ := ,/—07 — 03 and A’ := 07 + 03. Then, 


A'PEPU =cU so A'PEPU =cA'U. (65) 


Since the tangent to the great circle w3 = O is contained in w3-plane and is 
orthogonal to w, we can write (65) as 


A'Ps,PU = Ps, —Di,PU = cU and A' P35 PU = P% Dj, PU =A e 
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where D,. := (—@20; + @102)?. To summarize, we obtain 
/ 2. 
cAU = P,P) P (VU). (66) 


Generalizing the above calculation to arbitrary curves Sy C S? provides the A-term 
for the reconstruction operator in ELT [151]: 


Rert(g) = —P5, (D5, (8)). (67) 


Here, Di, is a second order differentiation in @+-plane along the tangential 
direction to the curve So, i.e. 


2 

Di, (g)(@.9):= Sye(o.y +50)| (68) 
with o denoting the unit tangent to Sp at @ € So. Analogous to (66), it is shown 
in [56, theorem 5.1]. that if So fulfills Orlov’s criterion, then Rg_r(g) recovers a 
singular pseudodifferential operator acting on Uirye (the cited theorem does not hold 
for single-axis data but the result is true also for that case as shown in [151]). When 
Orlov’s criterion is not fulfilled, then for single-axis data it turns out that the visible 
singularities of Ujye correspond to singularities of Re_r(g) whenever g = P(U true), 
see [151] and chapter > Microlocal Analysis in Tomography. 


Adaptation to ET 
The actual reconstruction operator in ELT is given as 


Reit(g) = Ps (1 = Ds, (g)). (69) 


This represents a local inversion method that recovers two important signal features. 
The pure A term defined in (67) emphasizes differences in data that occur at 
boundaries, i.e., this term picks up visible singularities. However, it does not 
distinguish interiors from exteriors since the derivative in these areas is typically 
small. The pure backprojection term (jz term) adds an averaged version of U that 
introduces contour to the reconstruction and allows one to distinguish objects from 
their surrounding. 

In the implementation of ELT, the Dy, operator in (69) is evaluated using a filter 
that is a smoothed version of the second derivative (a smoothed central second 
difference), and the half-width of the filter is determined by the signal-to-noise 
characteristics of the data. Hence, ELT has two regularization parameters, jz and 
the width of the derivative kernel. A method for choosing jz is given in [54] where 
the idea is to pick jz so that the grr reconstruction is as flat as possible inside a 
pre-specified feature. Furthermore, like in FBP, one can also convolve in the detector 
plane along the direction perpendicular to o (see (68)), which for single-axis tilting 
is just averaging over slices. 
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Fig. 6 A three-dimensional surface plot of the point response of FBP and ELT (wu = 0) for single 
axis tilting (beam direction is vertical, and the tilt-axis is perpendicular to the plane) with maximum 
tilt-angle is 60°. Point response is more localized in ELT (right) than in FBP (Jeff) (image from 
[155]) 


Comments and Discussion 

There are several advantages in using ELT as compared to FBP/WBP. First, let us 
compare the convolution kernels for ELT and FBP as done in [151, Figure 1]. The 
ELT kernel is local, so it is zero away from the origin whereas the FBP kernel is 
not. In fact, the oscillations of the FBP kernel on the interval where the ELT kernel 
is zero are about 7 % of the maximum amplitude. 

Next, ELT reconstruction operator has a more localized point response than FBP 
as shown in Fig.6. The “X”-like wings at the end of the angular range in both 
point responses are expected in any limited angle backprojection algorithm. The 
point response is however less localized in the FBP case, so the artifacts spread out 
causing a dilution of the actual signal relative to the background. This renders the 
FBP more noisy than the ELT and there is a higher chance to lose a weak signal 
using FBP. 


Generalized Ray Transform 

Illuminating larger fields of view (regions that extend about 8,000 nm from the 
optical axis) requires using wider electron beams. Then we must account for the 
fact that electrons, especially those farther from the central axis, travel along helical 
curves. Hence, the scattering operator is better modeled by (15) instead of (16), i.e., 
the ray transform P in (57) is replaced by a generalized ray transform that integrates 
over curves 5 +> y(s) that are solutions to (12). 

The FBP approach is extended in [106] to handle such generalized ray transforms 
that integrate over curved electron trajectories. This paper also considers alignment, 
which in this new setting is far more complex, see also [3, section 4.1.2]. In a 
similar manner, ELT has also been extended to the above setting. This involves 
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characterizing visible singularities analogous to theorem 1, and deriving local 
inversion methods [152, 154]. 


Approximative Inverse 


Overview 

The method of approximate inverse was originally developed in [118] for solving 
ill-posed integral equations of first kind. It has since then been extended to a more 
general setting. The starting point is, as with analytic methods, the reconstruction 
problem with continuum data (50). The idea is to derive a reconstruction kernel that 
is independent of data (so it can be pre-computed), and reconstruction is obtained by 
applying the kernel to data. The kernel is given as a solution of an adjoint equation 
that involves a mollifier. The regularizing property of approximate inverse comes 
from the choice of mollifier and robustness of the kernel against noise in data. 

The approximative inverse method is by now a well-established approach for 
regularization with applications to a wide range of inverse problems, see the survey 
by [170]. Recent developments include extending the framework to a Banach space 
setting and to handle certain nonlinearities in the forward problem [171, part V]. See 
also [115] for an interesting unification concept for general regularization methods 
based on approximate inverse. The application to ET is given in [101, 102]. 


A Priori Information 

As the name suggests, the original formulation of approximate inverse seeks 
provide an approximative inverse solution to the inverse problem. In this setting, 
the prior information is the same as for backprojection-based methods (section 
“Backprojection-Based Methods”). The approximate inverse method is in some 
sense the natural generalization of the FBP/WBP method to the setting where the 
forward operator is a general affine operator, like in (57). 


Data Pre-processing 
Computationally feasible implementations require an affine forward operator, which 
in particular includes operators of the type in (57). 


Reconstruction Operator 
We describe the reconstruction operator for linear operators in the Hilbert space 
setting and refer to [170] for its extensions to more general settings. 

Consider a linear forward operator 7: 2 — # between Hilbert spaces 2 and 
H¢, each with inner products (-, -) 9 and (-, -).v, respectively. Next, let Ey: R* x 
R? — C bea mollifier for 2°. The formal definition in the Hilbert space setting is 
given on [170, p. 26], but for our purposes it is enough to think of E,, as a smoothing 
operator that converges to a 6-distribution as y > 0: 


(Ey (x, Ug. > U(x) asy > OforallU € &. 
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Fix a point x and a mollifier and define the reconstruction kernel ¢ € W as the 
solution to the adjoint equation 


T* (¢%) = E,(x, -). (70) 


Note that the above calculation does not depend on data. Moreover, if the above 
equation lacks a solution, we consider the minimal norm solution, i.e., or is the 
element in 2° that minimizes the 2-norm of ¢ ++ T*(¢%) — E,(x,-). The 
reconstruction operator Raj: #7 —> 2 is now defined as 


Rai(g(x) = (8.9%) ~ forg ¢ H. (71) 


The rationale for (71) is that it provides a mollified solution to our inverse problem. 
To see this, let g = T (Upue), $0 


Rai(g)(x) = (g. Oo) = (TF Cease O!) 
= (Une: T*(0%)) , = (Unies Ey (x, ee 


Next, calculating the reconstruction kernel requires solving the adjoint equa- 
tion (70), which can be done in three different ways: (1) By means of an inversion 
formula for 7, (2) through a singular value decomposition, and (3) to use a 
projection method that recasts the adjoint equation into the finite dimensional setting 
where it can be solved using numerical methods. 

One issue is that the adjoint equation (70) needs to be solved for each evaluation 
point x. This is clearly unfeasible for 3D tomographic reconstruction problems. 
If the inverse problem is translation invariant, then it is natural to use convolution 
type mollifiers that only depend on the difference of the argument: E,(x,y) = 
e,(x — y). Given such mollifiers, the structure of the adjoint equation (70) does not 
depend on the point x: 


T*(@”) =ey. (72) 


Hence, the reconstruction kernel ¢” is independent of x and the reconstruction 
operator yields Rai(g) = Urue * €y Whenever g = T (Utrue)- 


Adaptation to ET 
The adaptation to ET lies in the computation of the reconstruction kernel which in 
turn depends on the mollifier and the forward operator. The choice of the mollifier 
should match the a priori information about Uy. and choice of y (regularization 
parameter) depends on the noise level in data. Since ET is a translation invariant 
inverse problem, the reconstruction kernel can be computed given these choices by 
solving (72). 

In ET we consider 7 as the linear part of (57), i.e., if A convolves with PSF 
in (57) then (72) becomes (Ps; o A*)(p”) = e,. We can now first compute v” 
by solving Ps, o v’ = e,, and then compute the reconstruction kernel ¢” from 
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A*(¢”) = v’. A common mollifier is to take e,(x) := y~!t(x/y) where t is a 
scaling function. PSF(@, -) has a closed form expression in Fourier space, which 
provides a closed form expression for the reconstruction kernel in Fourier space for 
parallel beam geometries relevant for ET. Hence, in ET the convolution with $” 
that defines the reconstruction operator is preferably computed in Fourier space, see 
[101, 102] for details. A special case is when the PSF(w,-) = 6,1 in (57), ie., 
we consider the inversion of the ray transform. The corresponding adjoint equation 
reads as Ps (@”) = ey, SO Rai(g) = Resp(g) where h in (60) given by ¢”. 


Comments and Discussion 

The benefit of approximative inverse, as compared to iterative (section “Iterative 
Methods with Early Stopping”) and variational (section “Variational Methods”) 
methods, lies in the combination of computational efficiency and flexibility regard- 
ing the type of forward operators it can handle, and to some extent, the type of a 
priori information it can encode. It is in fact easy, at a first glance, to underestimate 
the diversity of the a priori information that the approximate inverse method can 
encode. 

First, it is possible to work with a mollifier that is not rotation invariant [116, 
section 3]. This can be beneficial for ET since resolution is anisotropic due to the 
limited angle problem. Another possibility is given in [117] where approximative 
inverse is used for combined reconstruction and feature detection. The feature 
detection is here represented by a linear operator £: 2% — Y (Y is the Hilbert 
space of features). To compute an approximate inverse to C(Utnue) requires us to 
modify the adjoint equation (70) that provides the reconstruction kernel. In this 
new setting, it becomes 7* (x) = L( By (x, -)). If £ is translational invariant, 
the adjoint equation is again independent of the evaluation point x and it can be 
written as 7* (¢”) = L(e,). The case when C£ is an edge detector is treated in 
[117]. One can also select 7 = A and thereby perform A-tomography (section on 
p. 994), so approximate inverse extends ELT to the setting where one also includes 
deconvolution of the PSF. 

A drawback with the approximative inverse method is that it is difficult (if not 
impossible) to devise schemes for selecting the filter y that takes into account the 
specific stochasticity of the data. 


Iterative Methods with Early Stopping 


Iterative methods are a broad class of well-studied methods, see [11,50,77] for good 
surveys of both theory and implementation. The idea is to construct an iterative 
scheme in 2 that, in the limit, converges to a solution of (55), i.e., one of possibly 
infinitely many ML/least-squares solutions of (53). 


Overview 

There are three classes of iterative methods, conjugate gradient, Maximum- 
Likelihood Expectation Maximization (ML-EM), and iterative algebraic methods. 
These have over the years been extended in various ways, e.g., to handle nonlinear 
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forward operators, to account for different types of a priori information (like 
positivity), and to handle more complex noise models for data. Surveys of these, 
and other developments, are provided in [25, 81] for iterative algebraic techniques, 
in [92] and chapter > Iterative Solution Methods for conjugate gradient type of 
methods, and in chapter EM Algorithms for ML-EM type of methods. 

In the context of ET, only iterative algebraic methods are used. Hence, we focus 
on this category following the excellent exposition in [136, section 5.3], see also 
chapter > Tomography. The main idea is to split the inverse problem in (53) into 
a finite number of sub-problems. A cyclic iteration over these sub-problems is 
required to generate the next iterate. The specific choice of splitting leads to different 
iterative algebraic methods, such as the Algebraic Reconstruction Technique (ART) 
and the Simultaneous Iterative Reconstruction Technique (SIRT). The common 
theme is however to generate an iterative sequence that, in the limit, converges to a 
least-squares solution of (53). 


Data Pre-processing 
Same as for the approximate inverse method (section “Approximative Inverse”). 


A Priori Information 

Iterative methods with early stopping share a common a priori assumption, namely 
that the iterative scheme is semi-convergent. This means that initial iterates recover 
low-frequency components of the signal in (53). Now, assume noise from data 
mainly influences the high-frequency components of the signal. Then, an ill-posed 
problem can be regularized by stopping the iterates before too much of the data noise 
impairs the reconstruction, i.e., the number of iterates becomes a regularization 
parameter. Iterates are also often smoothed for further regularization. 

One can also account for additional a priori information. A common case is to 
require that Ujue is a positive function, or more generally, belongs to a convex set. 
Such constraints can be enforced by projecting iterates onto the said convex set, 
albeit at a significant computational cost. 


Reconstruction Operator 
Consider (53) and split the data (tilt-series) g € R” into N subsets: 


g=(g1--..8y) withg, €R” andm =m, +---+my. (73) 


Next, let t;:R” —> R” be the projection onto the j:th data component, so 
t;(g) := g;, and introduce the partial forward operator T ;: 2° > R”/ as 


T,U):=(t;°T)U) forj =1,...,N. 
Then (53) splits into the following set of N sub-problems: 


gj ‘= T; (aus) + ge where ge i (gre), (74) 
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The k:th iterate in an iterative algebraic method is obtained by performing an 
inner AN -step iteration: 


UP = Ux-1 
Uj = UJ" + w(T% oC;')\(g, -T) UZ), GF =L...N (75) 
U; = Ae 


Here, C;:R”/ — R”/ is a fixed linear positive definite operator, and a typical 
choice for linear T; is C; := Tj 0 T:. The parameter jz > 0 is a relaxation 
parameter that is needed for noisy problems, like the one in ET, to reduce the impact 
of noise in data and speed up convergence. 


Remark 22. There is a geometric interpretation of the iterates in (75). Each can be 
seen as the result of performing N consecutive projections in reconstruction space 
2. More precisely, the j-iteration in (75) can be written as U/ := a (U; ne 
where a (U) <= (1-—w)U + wa, (U) with aj): 2 — ZX denoting the projection 
operator 


nj(U):=U+(T}°0C;')(¢; -T;U)) forj =1,...,N. 


Note that when C ; := 7; 077, then z; (U) is merely the projection of the residual 
g; — 7 ;(U) onto the solution space of T ;(U) — g; = 0. 


The specific choice of splitting and choice of C ; determines the type of iterative 
algebraic method. The three most common ones are listed below: 


Algebraic Reconstruction Technique (ART): This is when data (73) is split up so 
that g; is a scalar corresponding to a single data point, ie., N = m, and 
C; =Tj0 T; . The iterative sequence in (75) is now easier to understand when 
expressed in the full discretized setting. The (fully discretized) linear forward 
operator is given by multiplication with a (m x n)-matrix whose rows are 
denoted by a; € R" for j = 1,...,m. The corresponding sub-problems (74) 
are now given as 


gj =a; -Utwe + g7" for 7 =1,...,m, 


and the projection calculation in (75) is expressible as 


Uj:=Ul +p (¢; —a;-Uj"')-a; for j =1,...,m. 


lla; lI? 
The ART, first introduced in [70], is perhaps the first iterative algebraic 
technique used for tomographic reconstruction. Later it was recognized it as 
a special case of Kaczmarz’s method. 
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Simultaneous Iterative Reconstruction Technique (SIRT): Compared to ART, 
SIRT is the other extreme in the sense that data is not slit, i.e., N = 1. Hence, 
SIRT reconstruction is given as 


Us = Ux + uT*(g¢ — T(Ux)). (76) 


Thus, each iterate corresponds to a full update of the signal that involves a 
complete sweep through all the data points. 
The SIRT was introduced to the scientific community in the context of 
tomography by [65] as an alternative to ART. It was later discovered that the 
SIRT is equivalent to Landweber iteration. 

Simultaneous ART (SART): SART, also called block iterative ART, was first 
introduced to the imaging community by [5]. It is a compromise between ART 
and SIRT in the sense that NV equals the number of views. For ET this means 
that NV is the number micrographs in the tilt-series, i.e., VN = my and g i€ Rice 
corresponds to the 7:th micrograph. 


The regularizing property of iterative algebraic techniques is far from resolved in the 
general setting. Most of the literature focuses on proving convergence properties, as 
reviewed in [25]. Such results are of less interest for ill-posed problems since iterates 
are stopped long before convergence (assuming the iterates do converge to some 
point). A fast convergence is nonetheless desirable since this would require fewer 
iterates before a “good” reconstruction is obtained. In general, the convergence of 
iterative algebraic methods depends on the splitting. For tomographic data, a good 
strategy is to do a splitting so that the directions are as orthogonal as possible to the 
previous ones. Curiously enough, a random choice of directions is almost as good 
as the optimal choice. This was also recently proved mathematically in [26]. 


Adaptation to ET 

The adaptation of iterative algebraic methods to ET lies in the choice of forward 
operator and its adjoint, and in how to select the number of iterates kya, and the 
relaxation parameter jL. 

In ET it is rather natural to split the tilt-series into micrographs (2D TEM 
images), so current implementations of iterative algebraic methods are based on 
SART rather than ART (even though this is not explicitly mentioned). Modern 
implementations also include possibility for more involved projection schemes, like 
component and block component averaging. 

As of writing, in the context of ET there is no theory that provides a practically 
useful criteria for how to choose the number of iterates ky ax. This is perhaps not 
that big of an issue since one can always run a couple of extra iterates. If one saves 
intermediate iterates, each such iterate corresponds to a specific choice of Kmax. 
Concerning 4, an analysis of how its choice influences the qualitative behavior 
of the iterates must exploit the specific structure of the forward operator. For the 
2D ray transform one can show that a strong under-relaxation, i.e., small jz results 
in iterations that first determine the smooth parts of the signal, while the higher 
resolution details appear later [136, pp. 113-115]. It is reasonable to expect that the 
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same holds for a forward operator of the type (57), so the common suggestion in ET 
is 4. = 0.01 (recommended range between 0 and 0.1). 

Yet another aspect is related to how the reconstruction space 2 is discretized in 
an implementation. The straight-forward approach is to simply evaluate the function 
at each voxel center. In [64, 109] one claims that a discretization based on Kaiser— 
Bessel window functions (blobs) improves the robustness against noise. 

Finally, we consider the issue of a priori information. A variety of a priori 
information relevant for ET can be incorporated into iterative algebraic methods. 
One can beforehand determine regions where the signal has to attain a given value 
(say zero), one can set upper and lower values for the signal, or only update parts of 
the signal. Such Projection Onto Convex Sets (POCS)-type of constraints, that can 
be formulated as the signal belonging to a convex set, can be enforced by projecting 
iterates. POCS was introduced to the electron microscopy community in [28], but 
convincing evidence of benefits is lacking in ET [145, p.315]. Iterative algebraic 
methods can also be used to solve variational regularization methods (section 
“Variational Methods”) with quadratic energy functionals. A final development 
is the so-called discrete ART where one discretizes the range of the allowed 
reconstructions. This is obviously suitable for specimens that consist of only a 
few different materials (gray levels), like in material sciences, and results are 
encouraging [13,72]. 


Comments and Discussion 

Iterative algebraic methods offer flexibility regarding the type of forward operators 
it can handle, and to some extent, the type of a priori information it can encode. 
Compared to approximate inverse (section “Approximative Inverse’”’), the flexibility 
is greater, but so is the computational burden. 

SIRT is considered to give smoother reconstructions, but converges more slowly. 
Still, it is now part of most software packages for ET in both life and material 
sciences. Both SART and SIRT perform better than FBP/WBP [120], even though 
the difference is not that dramatic. There are in fact few situations where the 
significantly longer run-time associated with SART/SIRT is worth the effort. The 
run-time of SART/SIRT is however becoming less of an issue thanks to recent 
parallel implementations that make use of hardware acceleration. These allow 
reconstructions within a reasonable time of regions as large as 2048 x 2048 x 512 
voxels from tilt-series of 60-80 micrographs, each with a size of 2048 x 2048 pixels 
[177]. We finally mention [72] where SIRT, total variation regularization (section 
on p. 1008), and the discrete algebraic reconstruction technique are compared for 
material sciences application of ET. 


Variational Methods 


Variational methods provide a flexible framework for regularization of (53). Here, 
the reconstruction is given as a solution to an optimization problem: 
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min pS(U) + D(TW), g) for g € R” givenas in (53). (77) 
VEX 


In the above, T: 2 — R” is the sampled forward operator, which for ET is given 
by (52), S: 2 — Ry is the regularization functional that enforces uniqueness and 
stability by incorporating a priori knowledge about Uime € 2, D:R” — Ry4 is 
the data discrepancy functional that quantifies the goodness-of-fit against measured 
data, and yz > O is a regularization parameter quantifying the compromise between 
accuracy and stability. Its choice should depend on the model for the noise, as well 
as on estimates of the size of the error. Formally, without knowledge of the latter, 
it is not possible to guarantee a result with low error, neither absolutely nor as a 
percentage [10]. 

If (77) is to be a regularization of (53), then solving (77) must be well-posed. 
Such issues form a central part of the general theory, and current state of the art is 
nicely surveyed by [169] in the context of imaging. 

In the fully discrete setting (54), there is also a nice connection to statistical 
regularization. If D is chosen as an affine transformation of the negative log- 
likelihood of the random variable modeling g, then (77) corresponds to a maximum 
a posteriori estimate where the prior probability distribution is given as a Gibbs 
measure with an energy S [91, subsection 3.3.2]. Extending this analogy to (53) is 
more elaborate since it requires a notion of Gibbs measure on the Hilbert/Banach 
space 2. 


Data Pre-processing 
Same as for the approximate inverse method (section “Approximative Inverse”). 


Entropy Regularization 


Overview 
Entropy regularization refers to the case when the regularization functional S in (77) 
is given by an entropy type of functional. There are plenty of examples, mainly from 
astronomy, where entropy type of functionals are used for regularization. 

Existence, uniqueness, stability, convergence, and convergence rate for the 
maximum entropy method are studied in [47] when the forward operator is a 
Fredholm integral operator, but most results seem to be valid for an arbitrary 
bounded linear operator from ! to a Hilbert space. A similar setting is dealt 
with in [49], which obtains stability, convergence, and convergence rate results. A 
wider class of regularization functionals is considered in [159], where convergence 
rates and error estimates are proved when the forward operator is a compact linear 
operator between Banach spaces, see also [137] for the case when the forward 
operator is given by convolution. Likewise, [82] establishes existence and regularity 
of solutions for convolution type of forward operators and a wide range of entropy 
type of regularization functionals. 

The application to ET is in [172] with a mathematical analysis in the language of 
regularization theory given in [165]. 
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A Priori Information 
It is clear that entropy type of functionals do penalize complexity in some sense, 
thereby acting as a regularization. Entropy type of functionals also measure 
information content [44], so many attempts to rationalize the usage of entropy for 
regularization are based on this principle. This motivation is artificial unless there is 
a natural probability distribution on 2°. This leads us to the framework of statistical 
regularization in which entropy is applied to the set of probability measures on 2 
[119, Method 3], but that is an entirely different approach for reconstruction. 
Another motivation is provided by [38, 174] that considers reconstruction meth- 
ods (called selection rules) for linear inverse problems with exact data. One starts 
off by postulating a set of axioms (consistency, distinctness, continuity, locality, and 
composition consistency) that a reconstruction method should satisfy. Next, it is 
proved that the least squares method is the only reconstruction method consistent 
with these axioms when the signal to be reconstructed is a real-valued function, 
having both negative and positive values, and entropy regularization is the only 
reconstruction method consistent when the signal to be reconstructed is positive. 


Reconstruction Operator 
The case we consider here makes use of the relative entropy, so we are given a fixed 
prior p € 2 and the regularization functional is 


S)(U) := / (uee)In (=) — U(x) + p(x)) dx forUe 2. (78) 
Q p(x) 


Then, U +> S,(U) is convex and S,(U) > 0. Note also that when U,p € 2 
represent probability measures, then S,(U) = 0 if and only if U = p almost 
everywhere and —S,(U) equals the Kullback—Leibler divergence. 


Adaptation to ET 

In [172] one regularizes the inverse problem in ET by (77) with (78) and a data 
discrepancy given by a Mahalanobis distance (section “Notion of Solution”). The 
forward operators are as in (57) and the regularization parameter is chosen according 
to the Morozov principle, so a user must supply an estimate of the data error. 

The Morozov principle for choosing the regularization parameter is not usable 
for highly noisy data since it requires unreasonably accurate estimates of the data 
error. Furthermore, a higher reconstruction resolution requires one to estimate the 
nuisance parameters. Thus, the approach in [172] is extended in [165], which uses 
an iterated regularization scheme (not to be confused with iterative regularization 
in section “Iterative Methods with Early Stopping”) that generates a sequence 
of regularization problems. Within this sequence, the estimate of the data error 
is “updated.” The nuisance parameters @ +» C(q@) and Co are recovered by 
intertwining the iterates that update the signal U € 2 with least-squares iterates 
that update the nuisance parameters and locality in tomographic data is accounted 
for by constant continuation (section on p. 985). To formulate the precise scheme, 
for given data g € R” and nuisance parameters c € VY (the space of nuisance 


1008 O. Oktem 


parameters), we introduce 


2 


inf,| 
VEX 

Hence, €min(g,c) is the smallest possible data error (it is zero for exact data). Also, 
define A, as a smoothing operator whose degree of smoothing is regulated by a 
parameter b > 0, it can, e.g., be a low-pass filter where b is the cut-off threshold in 
the Fourier space. Finally, 0 < 6 < 1 is the regularization parameter that governs 
the updating of the estimate of the data error. Then, for a user provided choice of 
regularization parameters 5,b > 0, the iterates (U;,c;) € 2 x V in [165] are 
defined recursively as 


cj = argmin .cy |r7@-. c)— g| ne (79) 
Ap (U;— if prior is to be updated, 

Ge (80) 
p if prior is not updated, 

€; °= €min(g,cj) + i(|Te;.<0 — e| - cnin(€s¢))) (81) 
argmin ye g Sp; (U) 

Uj = : (82) 
|TW,¢/) =2 i” < €j- 


Comments and Discussion 

Entropy type of regularization performs fairly well, especially on cryo-fixated 
in vitro specimens that contain isolated particles in aqueous environment. The idea 
of updating the estimate of the data error through (81) by choosing 0 < 6 < 1 
is much more robust than directly specifying an estimate for it in (82). On the 
other hand, one has to be careful in updating of the prior (80). If the prior is 
updated, reconstructions appear de-noised and de-speckled, but there is a significant 
risk of creating structures that tend to grow as iterates proceed and therefore are 
erroneously interpreted as true structures. If the prior is to be updated, it is highly 
recommended that one uses an edge preserving filter as A,. Finally, the nuisance 
parameters can equally well be estimated off-line by a least-squares approach, 
removing the need for (79). 


TV Type of Regularization 


Overview 
We use the term “TV type” of regularization for variational methods in which S 
in (77) is chosen as 
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1/p 
S(U) := (/, Ivu(s)|" dx) (83) 


Standard TV regularization is when p = 1, see chapters » Total Variation in Imag- 
ing and >» Numerical Methods and Applications in Total Variation Image Restora- 
tion for surveys of its role in imaging. The emphasis here is on its usage in ET. 


A Priori Information 
Standard TV regularization is based on the a priori assumption that the signal to be 
recovered has a sparse gradient, e.g., like a step function. 


Reconstruction Operator 

The reconstruction operator is given as the solution of (77) with S as in (83) and D 
a suitable data discrepancy functional (section “Notion of Solution”). If the forward 
operator is affine, and the data discrepancy is strictly convex, then (77) has a unique 
solution. 


Adaptation to ET 

The first usage of TV regularization in ET was [1]. The approach is specifically 
tailored towards single axis tilting data (section “Examples Relevant for ET”) 
since one performs a series of 2D reconstruction in slices orthogonal to the tilt- 
axis. The regularizing functional is an anisotropic variant of TV that accounts for 
the anisotropy in resolution due to limited angle data (section “Incomplete Data, 
Uniqueness and Stability”). Standard TV regularization has also been applied in 
[71] on STEM data of specimens from material sciences (nano-particles). Results 
are compared against SIRT and TV regularization performs significantly better. 

Both publications above that deal with TV regularization for ET have a number 
of weaknesses. First, no guidance is offered on how to select the regularization 
parameter. Next, both publications unnecessarily assume a forward operator given 
as the ray transform, so ET data needs to be pre-processed in the same way as 
for backprojection-based methods (section “Backprojection-Based Methods”). This 
is not an issue when an amplitude contrast model (section “Forward Operator for 
Amplitude Contrast Only”) is good enough for modeling micrograph contrast, like 
for incoherent STEM imaging in material sciences, but it is inadequate for ET 
based on HRTEM data. Finally, the data discrepancy functional D is the 2-norm, 
corresponding to the assumption of additive Gaussian noise whereas the actual 
noise is more complex. This is however probably not that big of an issue (paragraph 
on p. 976). 

The first two points raised above are in fact addressed in [164] that considers 
variational regularization of (53) with an affine forward operator of the form (57). 
The data discrepancy functional D in (77) is given by the 2-norm and the 
regularization functional S is of the form 


I/p 1/q 
stu =a{ f |vuc|"ax) +n [ |v|"ax) ; (84) 


1010 O. Oktem 


Hence, choosing p = 1| and x = O gives the usual TV regularization. The 
implementation works with a forward operator that models both amplitude and 
phase contrast for any parallel beam geometry. Nuisance parameters w +> C(@) and 
Co are estimated by least-squares from tilt-series data and locality in tomographic 
data is accounted for by constant continuation (section on p.985). The main 
scientific contribution of [164] is however a method for choosing the regularization 
parameter that is specifically designed for highly noisy ET data. 


Comments and Discussion 

Variational methods, like TV regularization, often perform very well in reducing 
speckle and noise if the choice of regularization functional correctly encodes 
some of the a priori regularity properties of Ujue. An issue is however their 
computational feasibility, and this is especially so for TV regularization in 3D 
imaging. The non-differentiability of the regularization functional in TV makes it 
difficult to directly use efficient gradient-based methods for solving (77). There are 
several iterative approaches for this purpose, see chapter » Numerical Methods and 
Applications in Total Variation Image Restoration for further details. This issue of 
computational feasibility is also related to choosing the regularization parameter. 
In contrast to iterative methods (section “Iterative Methods with Early Stopping’), 
for variational approaches each choice of regularization parameter requires a new 
iterative sequence. Hence, a critical part of computational feasibility is the ability to 
choose the regularization parameter. 

Another issue with TV regularization relates to the suitability of using the (83) 
with p = 1. This results in reconstructions that often have too narrow dynamic 
range and suffer from stair-casing. This might not be an issue for automated image 
analysis tasks, like segmentation, but biologists consider such reconstructions to 
have an “unnatural” appearance. As an alternative to standard TV regularization, one 
could consider using other TV-like energy (semi) norms, e.g., empirical tests show 
that using 1.1 < p < 1.5 in (84) gives reconstructions that compare favorably to TV 
(the case p = 1) regarding noise/speckle reduction, while gray-scale variations are 
recovered better. One can also consider Besov norms that are discretization-invariant 
and better at recovering smoothly varying parts of the signal [105]. An interesting 
approach is to consider the norm parameter p as a nuisance parameters and try 
to estimate it from data, like in [182] for the Besov norm. Another is to combine 
TV regularization with Bregman iterates and/or use higher-order TV methods to 
better recover smoothly varying parts of the signal and reduce stair-casing [14]. 
An example, [193] considers a variant of Dirichlet regularization that makes use of 
geometric information. This corresponds to choosing the regularization functional 
S in (77) as 


S(U) =f u(a(U)(x), e(U)(x))| V U(x)|? dx 


where o(U)(x) denotes the mean Gaussian curvature of the level-set surface of U 
at x, and u: R* — R, is a user specified function. The optimization problem is by 
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solving an L?-gradient flow using calculus of variations. The method is compared 
against SART, SIRT, and WBP on a tilt-series of a cryo-fixated in vitro specimen 
with isolated HIV virions in aqueous buffer. 

Finally, one may also consider using a norm in a variable Lebesgue space 
(Theorem 2.15 and 2.17 in [36]). Already in [18] it was suggested that in image 
reconstruction, a smoother image could be obtained by an interpolation technique 
that uses a variable exponent that decreases monotonically from two to one as VU 
increases, see comments in pp. 7-8 in [36]. This however leads to a non-convex 
functional that is computationally difficult to handle. One approach is then use 
A-tomography to determine the variable exponent. More precisely, consider the 
regularization functional 


S(U) := ‘| | V 7x) [POP ax (85) 
2 


where p: R+ — [1,2] is monotonically decreasing, e.g., p(t) = 2/(1 + 2r), and p 
is pre-computed from data. If the singularities of o are at the same location as Unue, 
then using (85) instead of (83) yields much smoother reconstructions in regions of 
moderate gradient and thus prevents stair-casing [33]. One can use ELT to calculate 
p and use microlocal analysis to characterize the singularities of p that coincide with 
those of Ute (section on p. 994). Albeit convex, the downside of using (85) is that 
it is not 1-homogenous, so multiplicative scaling might change the values of the 
regularization functional. 


Sparsity Promoting Regularization 


Overview 

As we have seen, sampling theory is at the heart of many image reconstruction algo- 
rithms for tomography, like FBP/WBP (section “Backprojection-Based Methods’). 
These methods make the assumption that noise is predominantly a high frequency 
phenomena, whereas relevant image features occupy low frequencies of the signal. 
Most signals are however sparse (compressible) is some suitable representation. It 
turns out that this a priori knowledge, if properly utilized, allows one to recover the 
signal from relatively few and/or highly noisy measurements. 

A straightforward way to take advantage of sparsity often leads to solving (77) 
with an 0-norm as regularization functional, which unfortunately is computationally 
unfeasible. Within geophysics and other scientific/engineering disciplines, it has 
been known since the late 1970s that one can use the computationally feasible 
1-norm instead of the 0-norm is certain cases. This observation was put into a 
formal mathematical setting in 2004 with the advent of a new sampling theory, 
compressed sensing, which puts precise conditions on a linear inverse problem 
for when the 1-norm on U +> p(U) in (77) gives the sparsest solution [27]. The 
field has since then exploded with several remarkable and far-reaching results, see 
chapter >» Compressive Sensing and the recent book [59] for an up-to-date survey. 
Below we focus on the role of sparsity in ET. 
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A Priori Information 

Let us start by formalizing the notion of sparsity. We say that p: 2 -— Y isa 
sparsifying map for Ure € 2 whenever p(Utrue) is “well approximated” by a low- 
dimensional subspace of Y (Y is here some suitable vector space). A common 
example is when there is a fixed set {f;}; C % (dictionary) such that U ~ >>; ai¢; 
with most a; ~ 0. In such case, Y is the space of sequences and p(U) := {a;};. 
Another example is p(U) := | V U|, which leads to TV regularization (section on 
p. 1008). 


Reconstruction Operator 

Assume that p is a sparsifying map for Ujue € 2% in (53) (or (54)). It is now natural 
to consider the most sparse ML solution (55) by solving (77) using a regularization 
functional given as the 0-norm of U b p(U). 

Much of the theoretical development in compressed sensing deals with when one 
can replace this 0-norm with a 1-norm, the latter being computationally feasible. 
Most of the mathematical results apply for linear finite dimensional reconstruction 
problems (54) where the noise in data is additive Gaussian. If the matrix representing 
the fully discretized forward operator (sensing matrix) fulfills certain mathematical 
criteria (restricted isometry property and coherence w.r.t. the sparsifying map), then 
the optimization below yields the sparsest solution to (54): 


min ||o(U)||, + 7) -—g | - for g € R” given as in (54). 


UeR" 


Adaptation to ET 

Ideas of applying compressed sensing to ET have been discussed in [17] for STEM 
data from material science specimens. The forward operator is the ray transform 
and the data discrepancy functional is based on additive Gaussian noise model. 
The paper discusses dictionaries, like blobs (p. 1005), but examples show TV 
regularization. 


Comments and Discussion 

A key step in using sparsity is to suggest a sparsifying map. There is a wide 
range of (possibly over-complete) dictionaries, both analytic and learned, to choose 
from for sparsely representing texture/gray scale and edge information in images 
[48, 163]. Which ones are suitable for a specific ET application? As an example, 
the sparsest representation of a molecule is simply given by listing its atoms and 
their positions, e.g., following the RCSB Protein Data Bank specification. The 
corresponding sparsifying map is however only applicable when the electrostatic 
potential (the real part of the signal) is resolved to a 3D resolution of about 0.1 nm, 
which is way beyond what is reachable in ET applications in biology. Still, there 
are several notions for describing 3D structural information in macromolecular 
assemblies that could be used in the design of dictionaries. As an example, tertiary 
and quaternary structure descriptions are resolvable in ET, but it is unclear what 
dictionaries to use for encoding such structural information. 
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The next key issue is to what extent we have a sensing matrix that fulfills 
the criteria for replacing the 0-norm with the 1-norm. Unfortunately, there are no 
efficient methods for determining whether a sensing matrix is coherent w.r.t. a 
dictionary and/or fulfills the restricted isometry property. As an example, it is an 
open problem to provide deterministic and explicit methods that yield matrices that 
fulfill the restricted isometry property. On the other hand, it is well known that a 
random matrix will satisfy the restricted isometry property with high probability if 
its entries are samples of any sub-gaussian distribution [39, section 1.4.4]. There are 
also results in this direction for certain deterministic sensing matrices [21], but none 
of these results are applicable to ET. 


Other Reconstruction Schemes 


Here we very briefly mention other reconstruction methods used in ET. In [2] 
one applies geometric tomography to ET on STEM data from material sciences 
specimens. A similar approach is discrete tomography that was applied in [29] to 
ET on TEM data from biological specimens. This did not bring any benefits, but the 
situation is different in material sciences [12], see also comments regarding discrete 
ART on p. 1005. 

Another approach deals with simultaneous reconstruction and segmentation 
within the framework of variational methods. Here, the Mumford—Shah functional 
can be used, see chapter » Mumford and Shah Model and Its Applications to Image 
Segmentation and Image Restoration for an extensive review. In [94] one applies 
this approach to simulated ET data, and results are encouraging. 

Finally, there is the possibility of making use of a priori shape information in the 
regularization (shape-based regularization). Shapes constitute the most important 
a priori knowledge that biologists use in their analysis of 3D reconstructions. A 
difficulty is that shapes are complex objects that are difficult to precisely describe as 
mathematical entities in a way feasible for computational treatment. Next, in biology 
no two objects have identical shape. Hence, the notion of shape must incorporate 
some form of statistics. The first approach to demonstrate the importance of shape 
information in the context of ET was taken in [69]. Here, variational regularization 
is used with a regularization functional that is based on a spatial prior that encodes 
local shape related geometric information. The interesting outcome of this attempt 
is that prescribing shape information on a small sub-set of the region of interest 
sharpens the entire reconstruction. An entirely different approach, that is much more 
robust against misspecification in a priori shape information, is based on shape 
metrics from the Large Deformation Diffeomorphic Metric Mapping (LDDM) 
framework in [194], see also chapter » Shape Spaces. The idea is that one has shape 
information given as a set of regions D := {(2;}j=1,...~4 C So with associated shape 
templates J := {1;}j=1,..,k where [;:.Q; — R is such that 


Utrve| 2; ~ ;.1; for some admissible diffeomorphism @; € G. 
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Here, G is the group of admissible diffeomorphisms that acts by the natural group 
action on the set 2 of potentials. The above simply encodes the assumption that 
shapes of certain substructures in the specimen, namely those contain in 92;, are 
known up to a deformation. Such information can be obtained from an initial 3D 
reconstruction, or by prior knowledge. Shape-based regularization is now defined 
as variational regularization (77) with S given as 


S(U) = Sreg(U ) + &Sshape(U ; 9,35, 1) (86) 


where S;eg encodes a priori regularity properties of Use and Sghape is the shape 
functional accounting for shape information given by 9 and 3: 


k 
Sshape(U; 9,3, w) = min Y [dane (Let) + Hs $1.0) -Ulo, |. 
j=l 


In the above, dshape is the shape metric given by LDDM [194] that measures the 
shape similarity between the shape template and Uiue. Hence, one combines an 
energy functional for regularity (like TV or Dirichlet energy) with a shape metric 
against a shape template [138]. Initial tests on 2D tomography (see Fig. 7) show that 
it is clear that a priori shape information, even imperfect, has a great potential in 
improving the reconstruction. 


10 Validation 


A serious obstacle in the development of computational methods for ET is the lack 
of proper validation tools. None of the reconstruction methods mentioned in this 
review has been validated mathematically (e.g., proof of convergence and error 
estimates) in a context that is relevant for ET. Moreover, there are no theoretical 
results that give bounds on the best possible “resolution,” partly because it is unclear 
what notion of resolution to use. Hence, validation must be based on ET data 
from phantoms (specimens whose structure/potential is precisely known). Physical 
phantoms, i.e., physical specimens with a precisely known structure, only exist in 
material sciences. Hence, for many ET application one is confined to simulated data. 
To avoid committing an “inverse crime,” both the simulator and phantom generator 
need to be as accurate as possible. This is nontrivial as explained in [166, 187]. 

Finally, there is the issue of defining objective criteria for evaluating the quality 
of a reconstruction. This is closely related to the problem of defining a notion of 
resolution. Statistical approaches based on defining figures-of-merit are outlined in 
[30], but this approach only works for validation against simulate data. 


Notion of Resolution 

Intuitively one would expect that resolution quantifies the size of the smallest 
features that can be reliably reconstructed. Due to formal non-uniqueness (section 
“Incomplete Data, Uniqueness and Stability”) in ET, one needs a notion of 
resolution for the visible wavefront set. This in turn requires precise control of the 
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Slightly noisy 2D parallel beam tomographic data of (b) from three directions, 
0°, 45°, and 90°. 


The phantom. Shape template. Shaped based regu- 
larization using (c). 


FBP reconstruction. Shape template (with Shaped based regu- 
smooth edges). larization using (f). 


Fig. 7 Test of 2D tomography with imperfect shape information. All reconstructions are based 
on data in (a). It is interesting to see how edge smoothness of the shape template affects the 
reconstruction. (a) Slightly noisy 2D parallel beam tomographic data of (b) from three directions, 
0°, 45°, and 90°. (b) The phantom. (c) Shape template. (d) Shape-based regularization using (c). 
(e) FBP reconstruction. (f) Shape template (with smooth edges). (g) Shape-based regularization 
using (f) 


constants in the Sobolev estimates that characterize the visible wavefront set. Some 
work in this direction is pursued in [153]. Secondly, it would also be of interest to 
have the corresponding characterization of visible wavefronts when the PSFs for 
the TEM optics and detector are included, and/or when on considers the full inverse 
scattering problem (i.e., the scattering operator is given by (6)). 

The notions of resolution used in ET are all essentially different ways of 
estimating the spectral signal-to-noise ratio [144]. A serious drawback is that these 
notions do not account for the main degrading factor, the influence of the shot noise 
(section “Characteristics of the Noise’). Thus, a notion of resolution in ET has to 
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Fig. 8 Surface rendered electrostatic potentials calculated from atomic resolution models of a 
CPMV (left), a bacteriophage T4 head (middle), and a TMV (right) particle. The CPMV virion has 
a diameter of 28 nm [112], T4 bacteriophage head has a “diameter” of about 80 nm [108], and a 
TMV virion is about 300 nm long and 18 nm in diameter [132]. Hence, the diameter of the CPMV 
is about 1/3 of the diameter of the bacteriophage T4 head and 1.5 times the diameter of the TMV 


include probabilistic concepts, see, e.g., [157, 180, 190] for ideas along these lines. 
To summarize, there is still no useful notion of resolution for ET. 


11 Examples 


This section shows results of different reconstructions methods applied to simulated 
and experimental single-axis tilt-series data, the latter courtesy of FEI]. The WBP 
and SIRT reconstructions are obtained using IMOD v. 4.3.7 [124], the approximate 
inverse reconstructions are from Holger Kohr based on [102], and the variational 
regularization reconstructions are by software from Hans Rullgard based on [164]. 


Regularization Parameters 

For WBP, it is the distance r (in pixels) of the radial reconstruction kernel in Fourier 
space before it is cut-off by a Gaussian with variance o. For SIRT it is the number 
of iterations. For approximate inverse, it is the width y of the Gaussian mollifier 
(paragraph on p. 999), and for variational methods, it is jz in (77). 

Tilt-series data is not processed. WBP and SIRT assume data are samples 
of the ray transform, whereas the variational and approximate inverse methods 
implement (57) in which the PSF models the optics and the detector in the standard 
phase contrast model (section on p. 967). 


Balls 


This is a simulated single-axis tilt-series generated using the TEM simulation 
software in [166] of a phantom consisting of 40 balls with different size and contrast 
embedded in aqueous buffer. 

Simulations represent a single-axis tilt-series acquired from a 300 keV con- 
ventional bright-field TEM. The tilt-angle ranges from —60° to 60° with one 
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3D phantom. 0° micrograph. 0° micrograph (no 
noise). 


with r = 1.10 stopped after 10 Approximate inverse 
and o = 0.1. iterations. with y = 30. 


(p = 1 in (1.83)) The case p = 1.5 in The case p = 2 in (1.83) 
with y = 800. (1.83) and pz = 800. and pz = 800. 


Fig. 9 Simulated data from a balls phantom. (a) shows a 2D cross-section of the phantom and (b) 
is the zero-tilt image, with (c) showing the noise-free version. The remaining images show the same 
2D cross section as in (a) through different reconstructions. (a) 3D phantom. (b) 0° micrograph. 
(c) 0° micrograph (no noise). (d) WBP with r = 1.10 and o = 0.1. (e) SIRT stopped after ten 
iterations. (f) Approximate inverse with y = 30. (g) TV (p = 1 in (83)) with 4p = 800. (h) The 
case p = 1.5 in (83) and x = 800. (i) The case p = 2 in (83) and ~ = 800 


1018 O. Oktem 


The 0.12° tilt micrograph, a 4096 x 4096 image in the aligned tilt-series. 


Region 1 with CPMV. Region 2 with bacterio- Region 3 with TMV. 
phage T4 head. 


Fig. 10 Summary of experimental tilt-series. Top image (a) shows the zero-tilt micrograph and 
bottom row (b)-(d) show the three extracted regions that contain the particles illustrated in Fig. 8. 
(a) The 0.12° tilt micrograph, a 4096 x 4096 image in the aligned tilt-series. (b) Region 1 with 
CPMV. (c) Region 2 with bacteriophage T4 head. (d) Region 3 with TMV (Data courtesy of http:// 
www.fei.com/) 


> 


Fig. 11 (continued) 2D slices through 3D reconstructions of the three regions of interest using 
different reconstruction methods. Images (a)-(f) are from region 1, (g)-() are from region 2, 
and (m)-(r) are from region 3. (a) WBP with r = 1.1 and o = 0.2. (b) SIRT stopped after 
ten iterations. (c) Approximate inverse with y = 42. (d) TV (p = 1 in (83)) with w = 1,500. (e) 
The case p = 1.5 in (83) and w = 1,500. (f) The case p = 2 in (83) and x = 1,500. (g) WBP 
with r = 1.2 and o = 0.4. (h) SIRT stopped after ten iterations. (i) Approximate inverse with 
y = 42. Gj) TV (p = 1 in (83)) with w = 1,500. (k) The case p = 1.5 in (83) and w = 1,500. 
(I) The case p = 2.0 in (83) and w = 1,500. (m) WBP with r = 1.01 and o = 0.5. (n) SIRT 
stopped after ten iterations. (0) Approximate inverse with y = 30. (p) TV (p = | in (83)) with 
fp = 1,500. (q) The case p = 1.5 in (83) and w = 1,500. (r) The case p = 2.0 in (83) and 
jt = 1,500 (Data courtesy of http://www.fei.com/) 
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with r = 1.1 and stopped after 10 Approximate inverse 
o = 0.2. iterations. with y = 42. 


(p = 1 in (1.83)) The case p = 1.5 in The case p = 2 in (1.83) 
with u = 1500. (1.83) and w = 1500. and yw = 1500. 
with r = 1.2 and stopped after 10 Approximate inverse 
a =04. iterations. with y = 42. 


The case p = 1.5 in The case p = 2.0 in 


(1.83) and pz = 1500. (1.83) and pz = 1500. 
with r = 1.01 stopped after 10 Approximate inverse 
and o = 0.5. iterations. with y = 30. 
= lin The case p = 1.5 in The case p = 2.0 in 
ae, —_ = i (1.83) and pw = 1500. (1.83) and wz = 1500. 


Fig. 11 (continued) 
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micrograph every second degree, i.e., 61 micrographs in total. The 3D region of 
interest (29 is a rectangular 210 x 250 x 40 voxel region, with 0.5 nm voxels. 
Magnification is 25,000x, defocus is 3 4m, f = 2.7 mm (focal length), C, = 
2.1 mm (spherical aberration), C. = 2.2 mm (chromatic aberration), aperture 
diameter is 40 jum, and a, = 0.1 mrad (condenser aperture angle). The detector has 
16 «um pixels, Cyain = 80, and detector response follows (56) with a = 0.7,b = 
0.2,c = 0.1,a = 10, and 6 = 40. Finally, the total dose is 6,000 e- /nm?, which 
corresponds to 40 e~ /pixel in each micrograph. 


Virions and Bacteriophages in Aqueous Buffer 


This is an experimental single-axis tilt series of a cryo-fixated in vitro specimen 
that contains a mixture of TMV, Keyhole Limpet Hemocyanin, CPMV, and 
T4 bacteriophage particles in aqueous buffer, see Fig. 8. 

The single-axis tilt-series was acquired by FEI using a 300 keV conventional 
bright-field TEM (FEI Titan Krios with a Falcon direct electron detector). It consists 
of 81 micrographs that were aligned using IMOD software [124]. There are three 
regions of interest listed below (voxel size is 0.4767 nm): 


Region 1: 512 x 256 x 200 containing mainly CPMV virions, see Fig. 10b for 
corresponding cut-out from the zero-tilt micrograph. 

Region 2: 512 x 256 x 350 containing mainly a single bacteriophage T4 head, see 
Fig. 10c for corresponding cut-out from the zero-tilt micrograph. 

Region 3: 512 x 256 x 350 containing mainly portion of a single TMV virion, see 
Fig. 10d for corresponding cut-out from the zero-tilt micrograph. 


Micrographs are acquired at 29,370 magnification using a defocus of 6.0 pm 
and a detector with a pixel size of 14 1m. The total dose is unknown. So are the 
parameters for the TEM electron optical elements and illumination (Figs.9, 10, 
and 11). 


12 Conclusion 


Much of the development in ET has circled around new improved and sample 
preparation. There has been tremendous progress in increasing the resolving power 
in electron optical elements, which in turn is important for imaging in material 
sciences. Current state-of-the-art TEMs have an optical resolving power of 0.05 nm 
[93], which is to be compared to the “size” of an isolated neutral atom that ranges 
between 0.03 and 0.3 nm. There are also strong indications that we are reaching the 
physical limits for the optical resolving power of an electron microscope, and that is 
not due to imperfections in optics but rather “noise” from the specimen [158]. Thus, 
it is reasonable to assume that the optical resolving power will not improve much 
beyond 0.05 nm. 


Mathematics of Electron Tomography 1021 


For ET in life sciences, the aforementioned development is not that relevant 
since the resolving power of the TEM optics does not limit the 3D “resolution.” 
Instead, the limitation here is due to the noise in image data, which in turn is due 
to the limited dose (section “The Dose Problem’). On the other hand, technological 
development regarding automation in sample preparation and data collection will 
have a big impact on ET in life sciences. TEMs are now stable enough to allow 
for automated recording of multiple tilt-series from multiple regions of interests 
within one or multiple specimens. Thus, the amount of available high-quality tilt- 
series is rapidly increasing and mathematics will have a key role in providing 
better 3D reconstructions as well as extracting useful information from such 3D 
reconstructions. 

The inverse problem in ET also contains open mathematical problems, e.g., there 
are open problems related to uniqueness and stability (section “Incomplete Data, 
Uniqueness, and Stability”). Furthermore, the design of regularization methods need 
to better account for the regularity/sparsity that a specimen poses. An example 
is the variants of TV type of regularization which need to be analyzed carefully 
from this viewpoint. Notions of sparsity and shape that are applicable to flexible 
molecular assemblies and/or subcellular structures is a central theme. Yet another 
central topic is the regularization parameter selection in problems with highly noisy 
data and/or complex noise models (Poisson and Gaussian, or correlated Poisson 
and Gaussian). Another area where much remains to be done is to mathematically 
analyze intertwined reconstruction schemes. These are approaches for handling 
nuisance parameters (paragraph on p. 989) in the context of classical regularization 
that often work surprisingly well. Finally, the high level of noise in ET indicates 
that the best framework for reconstruction is offered by statistical regularization. 
This framework is yet to be applied to ET in the sense that one recovers not only 
a single estimator but also a measure of the uncertainty. Statistical regularization is 
computationally demanding, so it has had limited applications to imaging problems. 
Nevertheless, it can be successfully applied to real 3D tomography problems [182], 
so the approach should also be applicable to ET. 
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Abstract 

This chapter discusses diffuse optical tomography. We present the origins of this 
method in terms of spectroscopic analysis of tissue using near-infrared light and 
its extension to an imaging modality. Models for light propagation at the macro- 
scopic and mesoscopic scale are developed from the radiative transfer equation 
(RTE). Both time- and frequency-domain systems are discussed. Some formal 
results based on Green’s function models are presented, and numerical methods 
are described based on discrete finite element method (FEM) models and a 
Bayesian framework for image reconstruction. Finally, some open questions are 
discussed. 


1 Introduction 


Optical imaging in general covers a wide range of topics. In this chapter, we mean 
techniques for indirect imaging using light as a method for obtaining observations 
of a subject. In a typical experiment, a highly scattering medium is illuminated by 
a narrow collimated beam, and the light that propagates through the medium is 
collected by an array of detectors. There are many variants of this basic scenario. 
For instance, the source may be pulsed or time harmonic, coherent, or incoherent, 
and the illumination may be spatially structured or multispectral. Likewise, the 
detector may be time or frequency resolved, polarization or phase sensitive, located 
in the near or far field, and so on. The inverse problem that is considered is to 
reconstruct the optical properties of the medium from boundary measurements. 
The mathematical formulation of the corresponding forward problem is dictated 
primarily by spatial scale, ranging from the Maxwell equations at the microscale 
to the radiative transport equation at the mesoscale and to the diffusion theory at 
the macroscale. In addition, experimental time scales vary from the femtosecond on 
which light pulses are generated, through the nanosecond on which diffuse waves 
propagate, to the millisecond scale on which biological activation takes place and 
still longer for pathophysiologic changes. 

In this chapter, we concentrate primarily on the macroscopic scale and the 
diffusion model for light propagation. The derivation of this model and its limits 
of applicability are discussed in section “Radiative Transfer Equation.” Historically, 
a large amount of early development considered analytic forms for the Green’s func- 
tion of the diffusion equation and series expressions for the effect of perturbations of 
these propagators by inhomogeneities; usually, only first-order linear methods were 
considered. These are discussed in section “Green’s Functions and the Robin to 
Neumann Map.” As computational methods become more readily available, more 
sophisticated approaches using optimization and Bayesian methods are becoming 
more accepted. We discuss these approaches in Sect. 4. 
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2 Background 


Figure | schematically illustrates the two main types of measurement system: time 
resolved and intensity modulated. In the former, a short duration pulse ~5—10 ps 
is employed, and in the latter, a steady-state intensity is created, modulated at a 
frequency in the range 100-—1,000 MHz. Obviously, the spectrum of frequencies 
in the time domain is many order higher than in the frequency-domain systems 
themselves, although the higher frequencies are very heavily damped and carry no 
information. A third domain is “DC” systems — these are the same as frequency 
domain, without the modulation. They are much simpler and cheaper, but without a 
complex wave, the inverse problem is nonunique [6]. 


Spectroscopic Measurements 


Attenuation of light in the near infrared (NIR) is due to absorption and scattering. 
The parameter of most interest is absorption which is caused by chromophores 
of variable concentration such as hemoglobin in its oxygenated and deoxygenated 
states. In the absence of scattering, the change in light intensity obeys the Beer— 
Lambert law 


= pad =a,[cld, (1) 


where d is the source-detector separation, which is equal to the optical path length, 
[c] is the concentration of chromophore c, and a, is the absorption coefficient per 
unit length per unit concentration of chromophore c and can usually be obtained in 
vitro. In the presence of scattering, the optical path length of transmitted photons 
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Fig. 1 Optical transillumination measurements made with a time-resolved system (/eft) or an 
intensity-modulated system (right) 
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follows a much more complex relationship. Hence, attenuation measurements alone 
do not allow quantification of chromophore concentration. 

Continuous intensity (DC) instruments measure changes in the intensity of light 
leaving the tissue surface [56]. This is frequently done in a purely spectroscopic 
manner, i.e., to obtain only global changes in chromophore concentration. In order 
to quantify concentration changes, additional information is required. One approach 
is to derive an approximate differential path length factor (DPF), which restores the 
approximate Beer—Lambert law for small changes in concentration 


— sin = DPF Spa d. (2) 


out 


Since there are typically several contributing chromophores, light of different 
wavelengths in the NIR region is employed, and regression techniques are used 
to find their relative weightings [26]. It was shown empirically [29] that the DPF 
is simply the mean time of light multiplied by the speed of light in the tissue. In 
fact, this relationship follows naturally from the diffusion approximation of light 
transport [8]. Furthermore, it is equally well approximated by the change in phase 
of an intensity-modulated system, at least at low modulation frequencies. 

Intensity-modulated measurements were first reported by [69]. Most systems use 
a heterodyne technique to mix the transmitted light with a reference beam of slightly 
different modulation frequency, thus producing a lower frequency envelope that is 
easier to detect using RF equipment. Time-resolved systems were first developed 
using a streak camera [29, 49, 73], an instrument with exceptionally high time 
resolution in the picosecond range but with high cost, relatively low dynamic range, 
and a significant inherent temporal nonlinearity due to a sinusoidal ramp voltage. 
Alternatively, time-correlation single photon counting (TCSPC) systems measure 
arrival times of individual photons by comparison with a reference pulse using a 
time-to-amplitude converter (TAC) device [20, 79, 83]. These systems have a high 
dynamic range and excellent temporal linearity. 


Imaging Systems 


Imaging methods can be divided into direct systems which seek to detect het- 
erogeneities in tissue by analyzing the transmitted (or, in some cases, reflected) 
light and indirect systems which attempt to solve the inverse problem of image 
reconstruction. The latter is the main emphasis of this article although the former is 
historically the precedent, in a similar manner in which x-ray radiographs were the 
precursor to x-ray computed tomography (CT). 

Transillumination of candle light for a patient suffering from hydrocephalus was 
reported as early as 1831, but the first significant attempt at diagnostic imaging using 
optical radiation was for breast lesions and was made by Cutler [27], who used a 
lamp held under the breast in a darkened room. However, even at this stage, multiple 
scattering effects caused a notable degradation in image quality. The recognition 
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of this fact led to many attempts to eliminate or minimize the degradation due to 
scattering ranging from collimation [55] and polarization discrimination [84] to 
coherence gating using holographic gating [90] or heterodyne detection [82]. 

With the introduction of time-resolved detectors came the natural attempt to 
use temporal gating to discriminate early arriving photons (which necessarily have 
the shortest optical path and therefore suffer the least number of scatterings) from 
later arriving photons which have undergone multiple scatterings and therefore have 
ill-determined photon paths. The early implementations of this idea used a Kerr 
gate as an ultrafast shutter [99]. However, this technique is limited to relatively 
low-scattering media due to the small dynamic range of the Kerr shutter. Other 
studies have been based on the streak camera [42] or TCSPC [19] systems described 
in section “Spectroscopic Measurements.” 

The attempt to physically discriminate between photons that have undergone 
different numbers of scattering events is inherently limited by the statistical 
likelihood of the low scattering number photons arriving at the detector. For the 
relatively optically thick tissues that are of interest in breast cancer screening or 
brain imaging, these photons are overwhelmed by noise. For this reason, indirect 
methods that solve an inverse problem based on recovering the spatially varying 
optical parameters that provide the best fit of a photon transport model with the 
measured data are becoming more prevalent. Within this framework, the three 
basic strategies (time resolved, intensity modulated, and DC systems) have all been 
developed and reported. In addition, many different geometrical arrangements have 
been investigated. Initial studies have been on 2D slice-by-slice imaging, although it 
is apparent that the photon propagation must in reality be described by a 3D model. 
Fully 3D methods are now appearing. 

In the remainder of this article, we will discuss the inverse problem and the 
strategies that have been adopted in order to solve it. In order to analyze this 
problem, we first have to consider the model of photon transport in dense media. 


3 Mathematical Modeling and Analysis 
Radiative Transfer Equation 


In optical imaging, light transport through a medium containing scattering particles 
is described by transport theory [54]. In transport theory, the particle conservation 
within a small volume element of phase space is investigated. The wave phe- 
nomenon of particles is ignored. The transport theory can be modeled through 
stochastic methods and deterministic methods. In the stochastic approach, individ- 
ual particle interactions are modeled as the particles are scattered and absorbed 
within the medium. The two stochastic methods that have been used in optical 
imaging are the Monte Carlo method and the random walk theory, of which two, 
the Monte Carlo is the most often used [5]. 

In deterministic approach, particle transport is described with integrodifferential 
equations which can be solved either analytically or numerically [5]. In optical 
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imaging, a widely accepted model for light transport is the radiative transport 
equation (RTE). The RTE is a one-speed approximation of the transport equation, 
and thus it basically assumes that the energy (or speed) of the particles does not 
change in collisions and that the refractive index is constant within the medium. For 
discussion of photon transport in medium with spatially varying refractive index, 
see, e.g., [16, 36, 62, 72]. 

Let Q C R”, n = 20r3 denote the physical domain where n is the dimension of 
the domain. The medium is considered isotropic in the sense that the probability of 
scattering between two directions depends only on the relative angle between those 
directions and not on an absolute direction. For discussion of light propagation in 
anisotropic medium, see, e.g., [44]. Furthermore, let dQ denote the boundary of the 
domain and § € S$"! denote a unit vector in the direction of interest. The RTE is 
written in time domain as 


1 a6(r.8) 


cot 


= i / @G-7) 0, 88s" + g(r, 58) 3) 
gn-l 


+5-VP(r,8) + (Us + Ua) (7, 8) 


and in frequency domain as 


(08) + 8- VOC) + (Us + Had OCS) 


= ps / OG )G(1.8)d8’ + g(r. 5), (4) 
gu-l 


where c is the speed of light in the medium, i is the imaginary unit, w is the angular 
modulation frequency of the input signal, and ws = [s(r) and yg = [a (7) are the 
scattering and absorption coefficients of the medium, respectively. The scattering 
coefficient represents the probability per unit length of a photon being scattered, 
and the absorption coefficient represents the probability per unit length of a photon 
being absorbed. Furthermore, ¢(r, 5) is the radiance, ©(S - 5’) is the scattering phase 
function, and q(r, 5) is the source inside Q. The radiance can be defined such that 
the amount of power transfer in the infinitesimal angle ds in direction 5 at time ¢ 
through an infinitesimal area dS is given by 


b(r, 8: t)3- DdSd3, 


where ?) is the normal to the surface dS [54]. The scattering phase function @(s - 5’) 
describes the probability that a photon with an initial direction Ss’ will have a 
direction § after a scattering event. In optical imaging, the most usual phase function 
for isotropic material is the Henyey—Greenstein scattering function [47] which is of 
the form 
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1 1—g? _ 2 
or 2x (1+g2—2g5+5/)’ : 
CT ae a (5) 


n = 3, 


4m (1492-298 09)?” 


where g is the scattering shape parameter that defines the shape of the probability 
density and it gets values between —1 < g < 1. With the value g = 0, the scattering 
probability density is a uniform distribution. For forward dominated scattering, g > 
0, and for backward dominated scattering, g < 0. The time-domain and frequency- 
domain representations of the RTE are related through Fourier transform. 

In order to obtain a unique solution for the RTE, the ingoing radiance distribution 
on the boundary 0Q, that is, (7,5) for S-b < 0, where D is the outward unit 
normal, needs to be known [24]. Several boundary conditions can be applied to the 
RTE [1, 33,57]. In optical imaging, the boundary condition which assumes that no 
photons travel in an inward direction at the boundary dQ is used [5] 


o(r,8)=0, reaQ, F-A<0. (6) 


This boundary condition, also known as the free surface boundary condition and 
the vacuum boundary condition, implies that once a photon escapes the domain 
Q, it does not reenter it. The boundary condition (6) can be modified to include a 
boundary source ¢o(r, S) at the source position e; C dQ, and it can be written in 
the form [96] 


do(r, S), re Ujé;, 5-1 <0 


a 2 (7) 
0, redQ\U;e;, Sn <0. 


o(r,s) = | 


In optical imaging, the measurable quantity is the exitance J,,(7) on the boundary 
of the domain. It is defined as [5] 


Jn (7) =f (S-d)P(7,5)ds,  r € dQ. (8) 
gn-l 


Diffusion Approximation 


In optical imaging, light propagation in tissues is usually modeled with the diffusion 
approximation (DA) to the RTE. The most typical approach to derive the DA from 
the RTE is to expand the radiance, the source term, and the phase function into series 
using the spherical harmonics and truncate the series [5, 24, 33]. If the spherical 
harmonics series is truncated at the Nth moment, Py approximation is obtained 
[5,33]. The first-order spherical harmonics approximation is referred to as the P; 
approximation, and the DA can be regarded as a special case for that. The most 
typical approach for utilizing the Py approximations in optical imaging has been to 
use them in angular discretization of the numerical solution of the RTE [14, 100]. 
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An alternative to the Py approximation is the Boltzmann hierarchy approach, 
in which moments of radiance are used to form a set of coupled equations that 
approximate the RTE [38]. Furthermore, the DA can be derived using asymptotic 
techniques [7, 17] leading to generalized diffusion equation or by using projection 
algebra [33,57]. If the speed of light is not constant, a diffusion equation with 
spatially varying indices of refraction can be derived [16]. 

Here, a short review of the derivation of the DA is given according to [54,57]. 
First, the P, approximation is derived, and then, the DA is formed as a special case 
for that. In the DA framework, the radiance is approximated by 


p(r,8) & OU) toe A): (9) 


1 
|s7- | ie 1 


where ®(r) and J(r) are the photon density and photon current which are defined as 


P(r) = I. o(r, 5)ds (10) 


J(r) = i; SH(r, 8)d5. (11) 
gn-l 


By inserting the approximation (9) and similar approximations written for the source 
term and phase function into Eq.4 and following the derivation in [5,54], the P, 
approximation is obtained 


(= + bs) (r) + V- I(r) = golr). (12) 
(= oo H,) Jin) + Vr) = a0), (13) 


where pu’, = (1 — g1) [Ms is the reduced scattering coefficient, go(r) and q)(r) are the 
isotropic and dipole components of the source, and g; is the mean of the cosine of 
the scattering angle [5,57] 


six f 6-3)06-5«6 (14) 


In the case of the Henyey—Greenstein scattering function, Eq.5, we have g; = g. 
To derive the diffusion approximation, it is further assumed that the light source 

is isotropic, thus g(r) = 0, and that “* J(r) = 0. The latter assumption, which in 

time-domain case is of the form 1 ase) _ = 0, is usually justified by specifying the 


condition la « ju), [5]. Utilizing these approximations, Eq. 13 gives the Fick’s law 


J(r) = —KkV G(r), (15) 
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where 


K=K(r) = (n (la + uy) (16) 


is the diffusion coefficient. Substituting Eq. 15 into Eq. 12, the frequency-domain 
version of the DA is obtained. It is of the form 


—V-KVO@(r) + Ua P(r) + a(n) = qo(r). (17) 


The DA has an analogue in time domain as well. It is of the form 


1d® 
—V-KVO@(r) + pg P(r) + . nn) = qo(r). (18) 


The time-domain and frequency-domain representations of the DA are related 
through Fourier transform, similarly as in the case of the RTE. 


Boundary Conditions for the DA 

The boundary condition (6) cannot be expressed in terms of variables of the 
diffusion approximation. Instead, there are a few boundary conditions that have 
been applied to the DA. The simplest boundary condition is the Dirichlet boundary 
condition which is also referred to as the zero-boundary condition. It sets the photon 
density to zero on the boundary; thus, ®(7) = 0, r € dQ [40, 87]. Alternatively, 
an extrapolated boundary condition can be used [33, 40, 87]. In the approach, the 
photon density is set to zero on an extrapolated boundary which is a virtual boundary 
outside the medium located at a certain distance from the real boundary. Both the 
zero-boundary condition and the extrapolated boundary condition are physically 
incorrect, and they have mostly been used because of their mathematical simplicity 
[40]. 

The most often used boundary condition in optical imaging is the Robin bound- 
ary condition which is also referred to as the partial current boundary condition 
[4, 24, 33, 40, 54, 87]. It can be derived as follows. Within the P; approximation 
framework (9), the total inward- and outward-directed photon fluxes at a point 
r € 0Q are 


J-(r) = -{ (S -A)P(r, 8)dS = yy P(r) — 5dr) (19) 
s-n<0 
I(r) = / (S-A)P(r, S)\dS = yy P(r) + 5o-J(0), (20) 
s-n>0 


where y,, is a dimension-dependent constant which obtains values y2 = 1/z and 
y3 = 1/4 [57]. To derive the Robin boundary condition for the DA, it is assumed 
that the total inward-directed photon flux on the boundary is zero; thus, 


J-(r)=0, rea. (21) 
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Utilizing Eq. 19 and the Fick’s law (15), the Robin boundary condition can be 
derived. It is of the form 


1 d®(r) - 


@(r) + —k—~ 0, redQ. (22) 
2yn ov 


The boundary condition (22) can be extended to include the reflection on the 
boundary that is caused by different refractive indices between the object and the 
surrounding medium. In that case, Eq. 21 is modified to the form 


J-(r)=RIT(r), 1 €aQ, (23) 


where R = R(x) is the reflection coefficient on the boundary dQ, withO < R <1 
[57]. Thus, if R = 0, no boundary reflection occurs, and Eq. 23 is reduced into 
Eq. 21. The parameter R can be derived from Fresnel’s law [40] or, if the refractive 
index of the surrounding medium is our = 1, by an experimental fit 


R = —1.4399n;,? + 0.7099n;,' + 0.6681 + 0.0636nin, (24) 
where nip is the refractive index of the medium [87]. Utilizing Eqs. 19 and 20 and the 


Fick’s law (15), the Robin boundary condition with mismatched refractive indices 
can be derived. It takes the form 


1 a®(r) _ 


Ko 


® 
Ook 2Yn av 


0, reag, (25) 


where € = (1 + R)/(1 — R), with ¢ = 1 in the case of no surface reflection. 
The boundary conditions of the DA for an interface between two highly scattering 
materials have been discussed, for example, in [4]. 

The exitance, Eq. 8, can be written utilizing Eqs. 19 and 20, the Fick’s law (15), 
and the boundary condition (25). In the DA framework, the exitance is of the form 


In(r) = IF (*)-J-(r) = 5) 


2 22) _ 2¥n P(r), rédaQ. (26) 
ay is 


Source Models for the DA 

In the DA framework, light sources are usually modeled by two approximate 
models, namely, the collimated source model and the diffuse source model. In the 
case of the collimated source model, the source is modeled as an isotropic point 
source 


qo(r) = 8(r — rs), (27) 
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where position r; is located at a depth 1/1’, below the source site [40, 87]. In the 
case of the diffuse source model, the source is modeled as an inward-directed diffuse 
boundary current J, at the source position ¢; C dQ [87]. In the case of the diffuse 
source model, Eq. 23 can be modified as 


J7(r) = RI*(*) + (U— Ry, 7 € Uje;. (28) 


Then, following the similar procedure as earlier, the Robin boundary condition with 
the diffuse source model is obtained. It is of the form 


1 .d® Ze Uya; 
RA 5 et CO es es 


Rn 29 
ap 0, r € 0Q\ U; &;. ( ) 


Validity of the DA 

The basic condition for the validity of the DA is that the angular distribution of the 
radiance is almost uniform. In order to achieve this, the medium must be scattering 
dominated; thus, “4, «< js. Most of the tissue types are highly scattering and the 
DA can be regarded as a good approximation for modeling light propagation within 
them. The DA has been found to describe light propagation with a good accuracy 
in situations in which its assumptions are valid [63, 80] and it has been successfully 
applied in many applications of optical tomography. 

However, the condition stating that the angular distribution of the radiance must 
be almost uniform is violated close to the highly collimated light sources. In 
addition, the condition cannot be fulfilled in strongly absorbing or low-scattering 
tissues such as the cerebrospinal fluid which surrounds the brain and fills the 
brain ventricles. Furthermore, in addition to the above conditions, the DA cannot 
accommodate realistic boundary conditions or discontinuities at interfaces. The 
diffusion theory has been found to fail in situations in which its approximations are 
not valid such as close to the sources [35,87] and within the low-scattering regions 
[37,48]. 


Numerical Solution Methods for the DA 
The analytical solutions of the RTE and its approximations are often restricted to 
certain specific geometries, and therefore, their exploitability in optical imaging is 
limited. Therefore, the equations describing light propagation are usually solved 
with numerical methods. The most often applied numerical methods are the finite 
difference method and the finite element method (FEM). The latter is generally 
regarded as more flexible when issues of implementing different boundary con- 
ditions and handling complex geometries are considered, and therefore, it is most 
often chosen as the method for solving equations governing light transport in tissues. 
The FE model for the time-varying DA was introduced in [9]. It was later 
extended to address the topics of boundary conditions and source models [80, 87] 
and the frequency-domain case of the DA [86]. It can be regarded as the most typical 
approach to numerically solve the DA. 
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Hybrid Approaches Utilizing the DA 


To overcome the limitations of the diffusion theory close to the light sources and 
within low-scattering and non-scattering regions, different hybrid approaches and 
approximate models have been developed. 

The hybrid Monte Carlo diffusion method was developed to overcome the 
limitations of the DA close to the light sources. In the approach, Monte Carlo 
simulation is combined with the diffusion theory. The method was introduced in [98] 
to describe light reflectance in a semi-infinite turbid medium, and it was extended 
for turbid slabs in [97]. In the hybrid Monte Carlo diffusion approach, Monte 
Carlo method is used to simulate light propagation close to the light source and 
the DA is analytically solved elsewhere in the domain. Monte Carlo is known to 
describe light propagation accurately. However, it has the disadvantage of requiring 
a long computation time. This has effects on computation times of the hybrid 
Monte Carlo approaches as well. A hybrid radiative transfer—diffusion model to 
describe light propagation in highly scattering medium was introduced in [93]. 
In the approach, light propagation is modeled with the RTE close to the light 
sources, and the DA is used elsewhere in the domain. The solution of the RTE 
is used to construct a Dirichlet boundary condition for the DA on a fictitious 
interface within the object. Both the RTE and the DA are numerically solved with 
the FEM. 

Different hybrid approaches and approximate models have been applied for 
highly scattering media with low-scattering and non-scattering regions. Methods 
that combine Monte Carlo simulation with diffusion theory have been applied 
for turbid media with low-scattering regions. The finite element approximation of 
the DA and the Monte Carlo simulation was combined in [41] to describe light 
propagation in a scattering medium with a low-scattering layer. However, also 
in this case, the approach suffers from the time-consuming nature of the Monte 
Carlo methods. Moreover, the hybrid Monte Carlo diffusion methods often require 
iterative mapping between the models which increases computation times even 
more. The radiosity—diffusion model [10,37] can be applied for highly scattering 
media with non-scattering regions. The method uses the FE solution of the DA to 
model light propagation within highly scattering regions and the radiosity model 
to model light propagation within non-scattering regions. A coupled transport and 
diffusion model was introduced in [18]. In the model, the transport and diffusion 
models are coupled, and iterative mapping between the models is used for the 
forward solution. Furthermore, a coupled radiative transfer equation and diffusion 
approximation model for optical tomography was introduced in [94] and extended 
for domains with low-scattering regions in [92]. In the approach, the RTE is used 
as the forward model in sub-domains in which the assumptions of the DA are not 
valid and the DA is used elsewhere in the domain. The RTE and DA are coupled 
through boundary conditions between the RTE and DA sub-domains and solved 
simultaneously using the FEM. 


Optical Imaging 1045 


Green’s Functions and the Robin to Neumann Map 


Some insight into light propagation in diffusive media can be gained by examin- 
ing infinite media. In particular, verification of optical scattering and absorption 
parameters is frequently made with source and detector fibers immersed in a 
large container and far from the container walls. In a finite domain, however, 
we will need to use boundary conditions. We will distinguish between solutions 
to the homogeneous equation with inhomogeneous boundary conditions and the 
inhomogeneous equation with homogeneous boundary conditions. In the latter case, 
we can use a Green’s function acting on go. In the former case, we use a Green’s 
function acting on a specified boundary function. 

We will use the notation Gg for the Green’s function for the inhomogeneous form 
of (18) with homogeneous boundary conditions and Gag for the Green’s function 
for the homogeneous form of (18) with inhomogeneous boundary conditions, i.e., 
we have Gg solving 


lo 
—V-«(r)VGo(r, r,t, t’) + (utr + -x) Ge(r,r’, t,t’) = (r’)5(t") (30) 
c 
rr e€Q\dQ,t>r 
IGea(taW t,t!) 


Ga(ra.r’, t,t’) + 26K (ra) ay 


0 (31) 


rq € OQ 


and Gag solving 


1 
—V-K(r)VGaa(r, Fs, t,t!) + (ao + “x) Gaa(r,¥s,t,t'!)=0 (32) 


réEQ\dQ,t >t’ 


IGaaQ (Fa. Fs, t,t’) 


= 8(r,)5(t") (33) 
ov 


Gaa(ra.Ts. t,t’) + 2K (ra) 


rs,%q € OQ. 


For a given Green’s function G, we define the corresponding Green’s operator as 
the integral transform with G as its kernel: 


ofi= | [oe .t0 fel yar 
—o0 JQ 


For the measureable, we define the boundary derivative operator as 
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Bi= me 


where appropriate we will use the simplifying notation 
G? = 5G 


to mean the result of taking the boundary data for a Green’s function. 

Since (18) is parabolic, we must not simultaneously specify both Dirichlet 
and Neumann boundary conditions on the whole of dQ. The same is true if we 
convert to the frequency domain and use a complex elliptic equation to describe 
the propagation of the Fourier transform of ®. Instead, we specify their linear 
combination through the Robin condition (25). Then, for any specified value g on 
dQ, we will get data y given by (26). The linear mapping Aq — y is termed the 
Robin to Neumann map and can be considered the result of a boundary derivative 
operator 6 acting on the Green’s operator with kernel Gag 


Arwlk, Had = BGaag. 


Since the Neumann data and Dirichlet data are related by (25), we may also define 
the Robin to Dirichlet map Arip(K, 4a) and specify the relationship 


Arw(k, Ha) — 20Arw(k, Ma) — I = 0 (34) 


The Forward Problem 


The Robin to Neumann map is a linear operator mapping boundary sources to 
boundary data. For the inverse problem, we have to consider a nonlinear mapping 
from the space of j4,, « coefficients to the boundary data. 

When considering an incoming flux J~ with corresponding boundary term q, the 
data is a function of one variable 


K 


Yq = Fa (‘“) (35) 


which gives the boundary data for the particular source term gq = D (J7 ). Using 
this notation, we consider the forward mapping for a finite number of sources 
{qj PS Aiaes Ss} as a parallel set of projections 


y=F (‘““) ; (36) 


K 
where 
F := (Fi,...,Fsy (37) 
(y1,.++,¥s) (38) 


tet 
il 
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We will consider (36) as a mapping from two continuous functions in solution 
space [la,K € X(Q) x X(Q) to continuous functions in data space y € Y(0Q). If 
the data is sampled as well (which is the case in practice), then F is sampled at a set 
of measurement positions {raj;i = 1,...M}. 

The inverse problem of diffusion-based optical tomography (DOT) is to deter- 
mine K, {4a from the values of y for all incoming boundary distributions q. If k, [a 
are found, we can determine jz) through (16). 


Schrédinger Form 


Problem (17) can be put into Schrédinger form using the Liouville transformation. 
We make the change of variables U = x!/?, by which (17) becomes 


=KV'@= 2079 VO (1 + ~) ®D = qo. 


Using 
V?U = «'PV72@ + 2V0-VK'? + OV! 
leads to 
Vee) +e otGey = OOO (39) 
«'/2(r) 
r € Q/dQ (40) 
dU (ra; @ 
Utes.) + 2x (ra) EE = c'(rsNg(rs:0) (41) 
ra € 0Q, (42) 
where 
= Ve ps io 
K1/2 KCK 
If k is real (i.c., @ = 0), there exist infinitely many «, j1a pairs with the same 


real k?, so that the measurement of DC data cannot allow the separable unique 
reconstruction of « and j2, [6]. For w 4 0, the unique determination of a complex 
k? should be possible by extension of the uniqueness theorem of Sylvester and 
Uhlmann [91]. From the complex k?, it is in principle possible to obtain separable 
reconstruction of first « from the imaginary part of k? and jz, from the real part; 
see [74] for further discussion. 

In a homogeneous medium, with constant optical parameters ,,K, we can 
simplify (42) to 


—V’O(r;@) +k’? O(r; @) = gu (ro), (43) 
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with the same boundary condition (25) and with 


ac + 1@ ro 
ga (SEE): gy - OE (44) 
CK K 


This equation is also seen directly from (17) for constant x. 
The solution in simple geometries is easily derived using the appropriate Green’s 
functions [8]. In an infinite medium, this is simply a spherical wave 


etklr—rs| 


®(r; w) = G(r, r.; @) = : (45) 


Ir —rg| 


where the notation G(r, r,; @) defines Green’s function for a source at position rg. 
Due to the real part of the wave number k, this wave is damped. This fact is the main 
reason that results from diffraction tomography are not always straightforwardly 
applicable in optical tomography. In particular, for the case @ = 0, the wave is 
wholly non-propagating. Even as w — oo, the imaginary part of the wave number 
never exceeds the real part. This is a simple consequence of the parabolic nature of 
the diffusion approximation. Although hyperbolic approximations can be made too, 
they do not ameliorate the situation. 


Perturbation Analysis 


An important tool in scattering problems in general is the approximation of the 
change in field due to a change in state, developed in a series based on known 
functions for the reference state. There are two common approaches which we now 
discuss. 


Born Approximation 
For the Born approximation, we assume that we have a reference state x» = 


(Lda, K)', with a corresponding wave ®, and that we want to find the scattered wave 
® due to a change in state x° = (a, 6)’. We have 


K=K+ 6, Mya=pPata. (46) 


Note that it is not necessary to assume that the initial state is homogeneous. 
Putting (46) into (17) gives 


—-V-(«K+ B)V®(r: w) + @ t+tat+ *) O(r; @) = qo(r; @) (47) 


with 


6=04 6. (48) 
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Equation 47 can be solved using the Green’s operator for the reference state 
= G[qgot+ V-pVO—-adl). (49) 


With Go, Green’s function for the reference state, we have 
O(r;@) = O(r:@) + / (Go(r, 1; @)V,- BF) V,. ®(r; @) — a(r’) &(r’; w)) dr’ 
Q 


= O(r;0) -{ (B(r')V,Go(r, 1; @) V(r; w) 
Q 
a(r’)Go(r, 1; @) P(r’; @)) , (50) 


where we used the divergence theorem and assumed B(ra) = 0; ra € OQ. 
If we define a “potential” as the differential operator 


V(a, B):=V- BV -a, (51) 
we can recognize (49) as a Dyson equation and write it in the form 
[I — GoV] ® = Gogo. (52) 
This may by formally solved by a Neumann series, 


Ton = Go + GoVGo + GoVGoVGo + > (53) 


or, equivalently, by using (48) in (50) to obtain the Born series 


® = @) a oe) +4 p?) Aoceales « (54) 
where 
60 — ® 
) = GVO 
&®” = GoVGoVO 
Rytov Approximation 


The Rytov approximation is derived by considering the logarithm of the field as a 
complex phase [54, 60]: 


O(r; w) = el) (55) 
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so that, in place of (48), we have 
In®@=Ind+u'. (56) 


Putting ® = e” into (17), we get 
>» 10) 
— OV - KVuy — Ok |Vuol|” + ® | wa + — ) = QQ. (57) 
c 


Putting ® = @e"” and (46) into (17), we get 


—BV - (« + BV (uy + w)— Oe + B) |V (uo + w)|? +.B (ua +a + %) = Qu. 
(58) 


Subtracting (57) from (58) and assuming ®& = © over the support of go, we get 
=K (2Vug- Vu? + [Vu |") — V - eWu! 
=V- BV (uo +) + BIV (uo +w)| — a. (59) 
We now make use of the relation 
V KV ® = OV - KV + 2«VO-Vu +O V - KV® (60) 
= O(V -«Vub + 20«Vug- Vu’) + WV - KV®. (61) 


The last term on the right is substituted from (17) to give 


V-KVw® 
Vee + ova ve = ——_— +u (1+ ~) ite) 


Substituting (62) into (59) and using 
®V - BVup + OB |Vuo|* = V - BV (Pup) 
we arrive at 
-V - «Vib + @ + ~) wd 
=V-BVO—-ab+ OV - PYW +« [Vu (63) 


The approximation comes in neglecting the last two terms on the right, which 
are second order in the small perturbation. The left-hand side is the unperturbed 
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Fig. 2 Top: absorption and scattering images used to generate complex fields. Disk diameter 
50mm, absorption range j4 € [0.01-0.04] mm, scatter range yp, € [1-4] mm—!. The complex 
field ® was calculated using a 2D FEM for a 6-function source on the boundary at the 3 o’clock 
position. A reference field ® was calculated for the same source and a homogeneous disk with 
fla = 0.025mm!, xf = 2mm!. Bottom: the difference in fields @ — ® (real and imaginary) 
and the difference of logs In ® — In ® (real and imaginary) 


operator, and so the formal solution for u°® is again obtained through Green’s 
operator with kernel Go. Thus, the first-order Rytov approximation becomes 


@) (r: w) 
8 (pe) — 
u'(r;@) = (eo) (64) 


O(r; 0) (B()V,Go(r, 1; @)- Vp (rw) 


a(r’)Go(r, 1; @) ®(r’; @)) . (65) 


The Rytov approximation is usually argued to be applicable for larger perturba- 
tions than the Born approximation, since the neglected terms are small as long as the 
gradient of the field is slowly varying. See [60] for a much more detailed discussion. 

Illustrations of the scattered field in the Born and Rytov formulations are shown 
in Fig. 2. Since in the frequency domain the field is complex, so is its logarithm. The 
real part corresponds to the log of the field amplitude and its imaginary part to the 
phase. From the images in Fig. 2, it is apparent that perturbations are more readily 
detected in amplitude and phase than in the field itself. This stems from the very 
high dynamic range of data acquired in optical tomography which in turn stems 
from the high attenuation and attendant damping. It is the primary motivation for 
the use of the Rytov approximation, despite the added complications. 
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Linearization 


Linearization is required either to formulate a linear reconstruction problem (i.e., 
assuming small perturbations on a known background) or as a step in an iterative 
approach to the nonlinear inverse problem. We will formulate this in the frequency 
domain. In addition, we may work with either the wave itself, which leads to the 
Born approximation, or its logarithm, which leads to the Rytov approximation. 


Linear Approximations 
In the Born approximation to the linearized problem, we assume that the difference 
in measured data is given just by the first term in the Born series (54) 


® (r;w) = ©” (ra) (66) 
=— i. (B')V Ge o(t. 1; @)- Vi O(r'; w)a(r’) Ge o(r, 1; @) P(r’; w)) d°r’. 


From (26), we obtain for a detector at position rg € dQ 
y(ra;@) = yo(ra;@) +f K" (rg, 1: @) Ae dr! (67) 
ds @) = yolTa; dT; : 
a 4 Ba’) 


where K,, is given by 


(68) 


B fis Js 
K, (tar; @) = ( Go ottae O)0r,@) ) 


V/G8 y(ra.0'3 @)- Vr P(r; w) 


The subscript g refers to the incoming flux that generates the boundary condition 
for the particular field ®. 

Since the Rytov approximation was derived by considering the change in the 
logarithm of the field, we have in place of (68) simply 


KM"'(ra, Yr’; w) = K, (ra, r’: w). (69) 


yo(ra; @) 


Assuming that we are given measured data g for a sufficient number of input 
fluxes, the linearized problem consists in solving for a, 6 from 


r= (9) 


where iC, is a linear operator with kernel given by (68) or (69) and 


y’=g-Yo, (71) 


where yo is the data that would arise from state xo. 
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We can now distinguish between a linearized approach to the static determination 
of x = xo + (a, B)' and a dynamic imaging problem that assumes a reference 
measurement go. In the former case, we assume that our model is sufficiently 
accurate to calculate yo. In the latter case, we use the reference measurement to 
solve 


y' = g-. (72) 


This in fact is where the majority of reported results with measured data are taken. 
By this mechanism, inconsistencies in the modeling of the forward problem (most 
notably using 2D instead of 3D) are minimized. However, for static or “absolute” 
imaging, we still require an accurate model, even for the linearized problem. 


Sensitivity Functions 
If we take g to be a 5-function at a source position r, € dQ, then ® is given by 


a Green’s function too, and we obtain the Photon Measurement Density Function 
(PMDF) 


(73) 


; ( G§ o(ra. 03 ©) Gag ot’. Fs; ©) 
p(ra. 0. ¥s3@) = 


Vi/G§ (ta. 5) Vi Gag ot’, Fs; @) 
with the Rytov form being 


1 


i 
FB Pa si): 
Gig (Ya, 1s; @) 


p(ra 1.153 @) = 


It is instructive to visualize the various p-functions which exhibit notable differences 
for jt, and x and between the Born and Rytov functions, as seen in Fig. 3. 
Clearly 


K, = : p(ra.¥', rs; @)q(rs)drs = G'q, 
Q 


where G’ is a linear operator with kernel p. 
We can also define the linearized Robin to Neumann map 


Diag (as K) (4) = a A(ra, Ys; w)q(rs)drs = Hq, 


where H is a linear operator with kernel H given by 


H(ra,rs) = - p' (rast, rs @) Gay d’r. 
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Fig. 3 Top row: sensitivity function p for [,; left to right: real, imaginary, amplitude, and phase; 
bottom row: the same functions for k 


Note that there are no equivalent Rytov forms. This is because the log of the 
Robin to Neumann map is not linear. 


Adjoint Field Method 


A key component in the development of a reconstruction algorithm is the use of the 
adjoint operator. The application of these methods in optical tomography has been 
discussed in detail by Natterer and coworkers [31,74]. 

Taking the adjoint of Kc, defines a mapping Y(dQ) > X(Q) x X(Q) 


K* 4b = K, (ta, 1; ©) b (ra; @) dS (74) 
IQ 


AB /. Or: 
= i Gooltars OCs)  \aqew)ds. (75) 
02 \VirGog o(ta. 1 @)- VP"; @) 


Consider the reciprocity relation 
—8 ri 
Go (ta. 0; ©) = —G* ao 0(¥, Fa @) (76) 
with G* 9g.9, Green’s function that solves the adjoint problem 


(0) 
—V-K(r)V G* 9g o(r, ras @) + (ao = *) G* sQ.o(t, tai @) = 0 (77) 


re Q\dQ 
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0G* 9a o(¥, Fai @) 


G* 9a. o(¥. a; @) + 20K (ra) ap 


= d(rg; a) 
(78) 


tq € aQ. 


Now, we can define a function W by applying the adjoint Green’s operator to the 
function b € Y(dQ) to give 


W(r:@) = -f Go otal: w)b(ra; w) dS (79) 
Q 
7 / G* s0.0(8. 84; @)b (ta; ) dS. (80) 
Q 


By using (78), we have that W solves 


~ Vcr VW(r@) + (ac 2 *) W(r@) =0 reaQ\aa (81) 
OW (ra; 
W(r4s 0) + 2 (rs) TE = bery:0) rye 92 (82) 
and therefore, K* , is given by 
..( sw 
K* gb = ee (83) 


Finally, we have an adjoint form for the PMDF (73) 


p(ta.r',rs3@) = ( (84) 


= 90,0(1" Fa; ©) Gag o(t’, Fs; @) 
—ViiG aqo(t’, ta @): Vi Gag ot’, Fs} @) 


Time-Domain Case 
In the time domain, we form the correlation of the propagated wave and the back- 
propagated residual. Equation 83 becomes 


~,_( fo -POVMa 
kK gD = (0 * yo(1)- Vw(t)de , (85) 


where W(t) is the solution to the adjoint equation 
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Fig. 4 Top row: time-window sensitivity function p for [a — left to right: time gates 500—1,000, 
1,500—2,000, 2,500—3,000, 3,500-4,000 ps. Bottom row: the same functions for 


(-<5 ~V-K(n)V + palt)) W(r,t)=0 reQ\IQ,re[0,T] (86) 
Wir,T)=0, reQ (87) 
OW (ra, t) 


W(rg, t) + 2fk (ra) = b(ra, t) rg € 0Q,t € [0, T]. (88) 


ov 


This is much more expensive, although it allows to apply temporal domain 
filters to optimize the effect of “early light” (that light that has undergone relatively 
few scattering events). In Fig. 4, the sensitivity functions over a sequence of time 
intervals are shown. Notice that the functions are more concentrated along the direct 
line of propagation for early times and become more spread out for later times. 


Light Propagation and Its Probabilistic Interpretation 


In time-domain systems, the source is a pulse in time which we express as a 6- 
function 
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4,000.0 6,000.0 8,000.0 


Time 


Fig. 5 Example of temporal response functions for different source-detector spacings. Circles 
represent measured data and dashed lines are the modeled data using a 3D finite element method. 
Each curve is normalized to a maximum of | (Data courtesy of E. Hillman and J. Hebden, 
University College London) 


q(ta,t) = q(ra)d(t), (89) 


where q (ra) is the source distribution on dQ. Furthermore, if the input light fiber is 
small, the spatial distribution can be considered a 6-function too, located at a source 
position r,; € dQ 


q(va,t) = 5(ta — Fs; )5(C). (90) 


For this model, the measured signal y(rg,t) is the impulse response (Green’s 
function) of the system, restricted to the boundary. When measured at a detector 
Ya; € OQ, it is found to be a unimodal positive function of t with exponential decay 
to zero as tf — oo. Some examples are shown in Fig. 5, showing measured data from 
the system described in [83] together with modeled data using a 3D finite element 
method. The function can be interpreted as a conditional probability density of the 
time of arrival of a photon, given the location of its arrival. 
Consider the Green’s functions for (18) in infinite space: 
|? 


brs 
Hal 4eG—t7) 


/ \\ _ / 
G(r, r’,t,t') = Gaye t>t. (91) 
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Equation 91 has the form of the Probability Density Function (PDF) for a lossy 
random walk; for a fixed point in time, the distribution is spatially a Gaussian; for 
a fixed point in space, the distribution in time shows a sharp rise followed by an 
asymptotically exponential decay. In the probabilistic interpretation, we assume 


G(r, t,r’,t’) 
—  —_—_ = Py (tl 92 
(Gr,tr,t)de 7 *" lr) ve 
as a conditional PDF in the sense that a photon that has arrived at a given point does 
so in time interval [t,t + 6¢] with probability Py ;(t|r)é¢. Furthermore, the absolute 
PDF for detecting a photon at point r at time ¢ is given by 


G(r, t,r',t') = Py y (r,t) = Py (vr) Py a (t\r). 


Here, P,/(r) is interpreted as the relative intensity /(r)/Jo of the detected number 
of photons relative to the input number. 

For most PDFs based on physical phenomena, there exists a Moment-Generating 
Function (MGF) 


M(s) = E[P(t)e"], (93) 


where E|.] is the expectation operator, whence the moments (around zero) are deter- 
mined by 


a" M(s) 


n ~~ 4 
= as” os 


s=0 
and in principle the PDF P(t) can be reconstructed via a Taylor series for its MGF 


n 
M(3) = mo + mys +++ Mq)—+ (95) 
However, explicit evaluation of this series is impractical. Furthermore, we may 
assume that only a small number of independent moments exist, in which case we 
reconstruct the series implicitly given only the first few moments. In the results 
presented here, only the first three moments 779, m , and mz are used. They have 
the physical interpretations 


mo Total intensity /(r) 

mM, : 

— Mean time <t>(r) 
mo 

m m : : 

ice (“) Variance time o7(r). 
mo mo 


In order to test the validity of the moment method to construct the time-varying 
solution, we created a finite element model of a 4x 7x4cm slab. The optical 
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Fig.6 An example of the output time-of-flight histograms at each of nine detectors on the 
transmission surface of a slab. Solid curves are computed from (95) using the zeroth, first, and 
second moments and implicit extrapolation; crosses are computed using finite differencing in time 
of the system (18) and 300 steps of size 10 ps 


parameters were set to an arbitrary heterogeneous distribution by adding a number 
of Gaussian blobs of different amplitude and spatial width in both wg and k to a 
background of j4, = 0.1 cm~!, « = 0.03 cm. A source was placed at the center of one 
face of the slab and nine detectors placed in a rectangular array on the opposite face 
of the slab. The time-of-flight histogram of transmitted photons at each detector was 
calculated in two ways: (1) by solving the time-dependent system (18) using a fully 
implicit finite differencing step in time (2) by solving for the moments mo, m1, m2 
using (94) and deriving the time-varying solution via (95). 

One case is shown in Fig.6. The comparison is virtually perfect despite the 
grossly heterogeneous nature of the example which precludes the exact specification 
of a Green’s function. The moment-based method is several hundred times faster. 


4 Numerical Methods and Case Examples 
Image Reconstruction in Optical Tomography 


Optical tomography is generally recognized as a nonlinear inverse problem; linear 
methods can certainly be applied (e.g., [85]) but are limited to the case of small 
perturbations on a known background. 

The following is an overview of the general approach: we construct a physically 
accurate model that describes the progress of photons from the source optodes 
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through the media and to the detector optodes. This is termed the forward problem 
(section “The Forward Problem’). This model is parameterized by the spatial 
distribution of scattering and absorption properties in the media. We adjust these 
properties iteratively until the predicted measurements from the forward problem 
match the physical measurements from the device. This is termed the inverse 
problem. 

The model-based approach is predicated on the implicit assumption that there 
exists in principle an “exact model” given by the physical description of the 
problem, and the task is to develop a computational technique that matches 
measured data within an accuracy below the expected level of measurement error. 
In other words, we assume that model inaccuracies are insignificant with respect 
to experimental errors. However, the computational effort in constructing a forward 
model of sufficient accuracy can be prohibitive at best. In addition, the physical 
model may have shortcomings that lead to the data being outside the range of the 
forward model (nonexistence of solution). 

In the approximation error method [57], we abandon the need to produce 
an “exact model.” Instead, we attempt to determine the statistical properties of 
the modeling errors and compensate for them by incorporating them into the 
image reconstruction using a Bayesian approach (section “Approximation Error 
Approach’’). The steps involved in using the approximation error theory are (1) the 
construction and sampling of a prior, (2) the construction of a mapping between 
coarse and fine models, (3) calculation of forward data from the samples on both 
coarse and fine models, and (4) statistical estimation of the mean and covariance of 
the differences between data from the coarse and fine models. In [11], this technique 
was shown to result in reconstructed images using a relatively inaccurate forward 
model that were of almost equal quality to those using a more accurate forward 
model; the increase in computational efficiency was an order of magnitude. 

Reconstruction from optical measurements is difficult because it is fundamen- 
tally ill posed. We usually aim to reconstruct a larger number of voxels than there 
are measurements (which results in nonuniqueness), and the presence of noisy 
measurements can result in an exponential growth in the image artifacts. In order to 
stabilize the reconstruction, regularization is required. Whereas such regularization 
is often based on ad hoc considerations for improving image quality, the Bayesian 
approach provides a rigorous framework in which the reconstructed images are 
chosen to belong to a distribution with principled characteristics (the prior). 


Bayesian Framework for Inverse Optical Tomography Problem 


In the Bayesian framework for inverse problems, all unknowns are treated and 
modeled as random variables [57]. The measurements are often considered as 
random variables also in non-Bayesian framework for inverse problems. The actual 
modeling of the measurements as random variables is, however, often implicit, 
which is most manifest when least squares functionals are involved in the formu- 
lation of the problem. In the Bayesian framework, however, both the measurements 
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and the unknowns are explicitly modeled as random variables. The construction of 
the likelihood (observation) models and the prior models is the starting point of the 
Bayesian approach to (inverse) problems. 

Once the probabilistic models for the unknowns and the measurement process 
have been constructed, the posterior distribution 2 (x | y) is formed, which distribu- 
tion reflects the uncertainty of the interesting unknowns x given the measurements 
y. This distribution can then be explored to answer all questions which can be 
expressed in terms of probabilities. For general discussion of Bayesian inference, 
see, for example, [21]. 

Bayesian inverse problems are a special class of problems in Bayesian inference. 
Usually, the dimension of a feasible representation of the unknowns is significantly 
larger than the number of measurements, and thus, for example, a maximum 
likelihood estimate is either impossible or extremely unstable to compute. In 
addition to the instability, the variances of the likelihood model are almost invariably 
much smaller than the variances of the prior models. The posterior distribution is 
often extremely narrow and, in addition, may be a nonlinear manifold. 


Bayesian Formulation for the Inverse Problem 

In the following, we denote the unknowns with the vector x, the measurements with 
y, and all probability distributions (densities) by 2. Typically, we would have x = 
(LLa, ts), With jg and jt; identified with the coordinates in the used representations. 

The complete statistical information of all the random variables is given by 
the joint distribution z(x, y). This distribution expresses all the uncertainty of the 
random variables. Once the measurements y have been obtained, the uncertainty 
in the unknowns x is (usually) reduced. The measurements are now reduced from 
random variables to numbers, and the uncertainty of x is expressed as the posterior 
distribution z(x | y). This distribution contains all information on the uncertainty of 
the unknowns x when the information on measurements y is utilized. 

The conditional distribution of the measurements given the unknown is called the 
likelihood distribution and is denoted by z(y | x). The marginal distribution of the 
unknown is called the prior (distribution) and is denoted by 2(x). By the definition 
of conditional probability, we have 


m(x,y) = W(y|x)a(x) = a(x] y)x(y). (96) 


Furthermore, the marginal distributions can be obtained by marginalizing (inte- 
grating) over the remaining variables, that is, m(x) = f 2(x,y)y and x(y) = 
f (x, y) x. The following rearrangement is called Bayes’ theorem 


n(x |y) = x(y) ‘(vy | x)a(x). (97) 


If we were given the joint distribution, we could simply use the above definitions 
to compute the posterior distribution. Unfortunately, the joint distribution is prac- 
tically never available in the first place. However, it turns out that in many cases, 
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the derivation of the likelihood density is a straightforward task. Also, a feasible 
probabilistic model for the unknown can often be obtained. Then, one can use 
Bayes’ theorem to obtain the posterior distribution. The demarcation between the 
Bayesian and frequentist paradigms is that, here, the posterior is obtained by using 
a (prior) model for the distribution of the unknown rather than the marginal density, 
which cannot be computed since the joint distribution is not available in the first 
place. We stress that all distributions have to be interpreted as models. 


Inference 
Point estimates are the Bayesian counterpart of the “solutions” suggested by regular- 
ization methods. The most common point estimates are the maximum a posteriori 
estimate (MAP) and the conditional mean estimate (CM). Let the unknowns and 
measurements be the finite-dimensional random vectors x € RY, y ¢ R™. 

The computation of the MAP estimate is an optimization problem, while the 
computation of the CM estimate is an integration problem: 


Xmap = Sol max (x | y) (98) 


eee ery / xx(x|y)x, (99) 


where sol reads as “solution of” the maximization problem, E(-) denotes expecta- 
tion, and the integral in (99) is an N-tuple integral. 
The most common estimate of spread is the conditional covariance 


aS / (x —E@ | y))(@ — E(x |y))" aly) x. (100) 


Here, I’, | , is an N x N matrix and the integral (100) refers to a matrix of associated 
integrals. 

Often, the marginal distributions of single variables are also of interest. These 
are formally obtained by integrating over all other variables 


Re / x(x |y)%~. (iol) 


xe 


where the notation (-)—¢ refers to all components excluding the th component. 
Note that (x; | y) is a function of a single variable and can be visualized by 
plotting. The credibility intervals are the Bayesian counterpart to the frequentist 
confidence intervals, but the interpretation is different. The p%-credibility interval 
is a subset which contains p% of the probability mass of the posterior distribution. 


Likelihood and Prior Models 

The likelihood model z(y | x) consists of modeling the forward problems and the 
related observational errors. In the likelihood model, all unknowns are treated as 
fixed. The most common likelihood model is based on the additive error model 
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y=F(x)+e, 


where e is the additive error term with distribution z,(e) which is usually modeled 
as mutually independent with x. In this case, we can get rid of the unknown additive 
error term by pre-marginalizing over it. We have formally m(y|x,e) = d(y — 
F(x) + e) and using the Bayes theorem 


m(y |x) = me(y — F(x). 


For more general derivation and other likelihood models, see, for example, [57]. 
In the special case of Gaussian additive errors 2.(e) = NV (ex, .), we get 


1 
n(y |x) exp (= Ibe - Fle) — enIP), 


where T>! = LIL,. In the very special case of m-(e) = N(0,y7/), we of 
course get the ordinary least squares functional for the posterior potential. This 
particular model, however, should always be subjected to assessment since it usually 
corresponds to an idealized measurement system. 

For prior models (x) for the unknowns whose physical interpretation is a 
distributed parameter, Markov random fields are a common choice. The most 
common type is an improper prior model of the form 


a(x) exp Gaac - x.)IP) , (102) 


where L, is derived from a spatial differential operator. For example, || L.(x—x.)]|| 
might be a discrete approximation for Ie | Ax()|? d7. Such improper prior models 
may work well technically since the null space of L,, is usually such that it is annihi- 
lated in the posterior model. It must be noted, however, that there are constructions 
that yield proper prior models [57,59, 67]. These are needed, for example, for the 
construction of approximation error models discussed in section “Approximation 
Error Approach.” Moreover, structural information related to inhomogeneities and 
anisotropicity of smoothness can be decoded in these models [59]. 


Nonstationary Problems 
Inverse problems in which the unknowns are time varying are referred to as 
nonstationary inverse problems [57]. These problems are also naturally cast in 
the Bayesian framework. Nonstationary inverse problems are usually written as 
evolution—observation models in which the evolution of the unknown is typically 
modeled as a stochastic process. The related algorithms are sequential and in the 
most general form are of the Markov chain Monte Carlo type [32]. However, the 
most commonly used algorithms are based on the Kalman recursions [3,57, 61]. 

A suitable statistical framework for dealing with unknowns that are modeled with 
stochastic processes and which are observed either directly or indirectly is the state 
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estimation framework. In this formalism, the unknown is referred to as the state 
variable, or simply the state. For treatises on state estimation and Kalman filtering 
theory in general, see, for example, [3,34]. For the general nonlinear non-Gaussian 
treatment, see [32], and for state estimation with inverse problems, see [57]. 

The general discrete time state space representation of a dynamical system is of 
the form 


Xe+1 = Fre(Xe, We) (103) 
Ve = Ak (Xk, Ve), (104) 


where wy; is the state noise process, vx is the observation noise process, and (103) 
and (104) are the evolution model and observation model, respectively. Here, the 
evolution model replaces the prior model in stationary inverse problems, while the 
observation model is usually the same as the (stationary) likelihood model. We do 
not state the exact assumptions here, since the assumptions may vary somewhat 
resulting in different variations of Kalman recursions; see, for example, [3]. It 
suffices here to state that the sequences of mappings F, and A; are assumed to 
be known and that the state and observation noise processes are temporally uncorre- 
lated and that their (second-order, possibly time-varying) statistics are known. With 
these assumptions, the state process is a first-order Markov process. The first-order 
Markov property facilitates recursive algorithms for the state estimation problem. 
The Kalman recursions were first derived in [61]. 

Formally, the state estimation problem is to compute the distribution of a state 
variable x, € R% given a set of observations y ae R™, 7 © Z where Z is a set 
of time indices. In particular, the aim is to compute the related conditional means 
and covariances. Usually, Z is a contiguous set of indices and we denote Yr = 


(Vis--++ Ye): 
We can then state the following common state estimation problems: 


¢ Prediction. Compute the conditional distribution of x, given Y?, k > £. 
¢ Filtering. Compute the conditional distribution of x, given Y;, k = €. 
* Smoothing. Compute the conditional distribution of x; given Y;, k < £. 


The solution of the state estimation problems in linear Gaussian cases is usually 
carried out by employing the Kalman filtering or smoothing algorithms that are 
based on Kalman filtering. These are recursive algorithms and may be either real- 
time, online, or batch-type algorithms. 

In nonlinear and/or non-Gaussian cases, extended Kalman filtering (EKF) vari- 
ants are usually employed. The EKF algorithms form a family of estimators that 
do not possess any optimality properties. For many problems, however, the EKF 
algorithms provide feasible state estimates. For EKF algorithms, see, for example, 
[3, 57]. Since the observation models with optical tomography are nonlinear, 
the EKF algorithms are a natural choice for nonstationary DOT problems; see 
[30, 66, 81]. 
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The idea in extended Kalman filters is straightforward: the nonlinear mappings 
are approximated with the affine mappings given by the first two terms of the Taylor 
expansion. The version of extended Kalman filter that is most commonly used is 
the local linearization version, in which version the mappings are linearized at the 
best currently available state estimates, either the predicted or the filtered state. This 
necessitates the recomputation of the Jacobians 0A,/0x; at each time instant. 

The EKF recursions take the form 


Xkjke—1 = Fei (%k—1\k—1) + Sk—-1 + Br-1(Ux-1) (105) 
Veet = Im T eye a) + (106) 
Ky = Vetta? (Sag Peta” + Py) (107) 
Vie = (1 — Ki Jay) Veta (108) 
Xie = Xkje—1 + Ke (ve — Ac (xete—1)) (109) 


where x;\¢-; and x,\, are the prediction and filtering estimates, respectively, 
and Tyj,-; and Ty), are the approximate prediction and filtering covariances, 
respectively. Note that the Jacobian mappings (linearizations) are needed only in 
the computation of the covariances and the Kalman gain K;. 

The applications of EKF algorithms to nonstationary DOT problems have been 
considered in [30, 66, 81]. In [66], a random walk evolution model was constructed 
and used for tracking of targets in a cylindrical tank geometry. In [81], a cortical 
mapping problem was considered, in which the evolution model was augmented to 
include auxiliary periodic processes to allow for separation of cyclical phenomena 
from evoked responses. In [30], an elaborate physiological model was added to that 
of [81] to form the evolution model. 


Approximation Error Approach 

The approximation error approach was introduced in [57,58] originally to handle 
pure model reduction errors. For example, in electrical impedance (resistance) 
tomography (EIT, ERT) and deconvolution problems, it was shown that signif- 
icant model reduction is possible without essentially sacrificing the quality of 
estimates. With model reduction, we mean that very low-dimensional finite element 
approximations can be used for the forward problem. The approximation error 
approach relies heavily on the Bayesian framework of inverse problems, since the 
approximation and modeling errors are modeled as additive errors over the prior 
model. 

In this following, we discuss the approximation error approach in a setting in 
which one distributed parameter is of interest, while another one is not, and there 
are additional uncertainties that are related, for example, to unknown boundary data. 
In addition, we formulate the problem to take into account model reduction errors. 
In the case of optical tomography, this would mean using very approximate forward 
solvers, for example. 
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Let now the unknowns be (lq, [4s, §,e@), where e represents additive errors and 
€ represents auxiliary uncertainties, such as unknown boundary data, and jt, is of 
interest only. Let 


y = A(a, fs, €) +e € R” 


denote an accurate model for the relation between the measurements and the 
unknowns. 

In the approximation error approach, we proceed as follows. Instead of using 
the accurate forward model (jig, [ls,&) 1 A(a, ls, €) with (fq, Ws, €) as the 
unknowns, we fix the random variables (w5,&) <— ({s,9, &) and use a computa- 
tionally (possibly drastically reduced) approximate model 


Ma +> A(Ma, Ms, &0)- 


Thus, we write the measurement model in the form 


y = Alia, Us, €) te (110) 
= A(Ma, Ms,0, §0) + (A(Ha. Ms, &) — A(Mas Ms,0, £0) +e (111) 
= A(a, Ms, &0) +E +e, (112) 


where we define the approximation error & = Q([aq, fs,§) = A(Ua: bs, &) — 
A({a, [4s.0, 0). Thus, the approximation error is the discrepancy of predictions 
of the measurements (given the unknowns) when using the accurate model 
A([a, [ts, €) and the approximate model A({lq, /s.0. £0). 

Using the Bayes’ formula repeatedly, it can be shown that 


m(y|x) = [0 = Al 1.0080) = 2) (12) de (113) 


since e and x are mutually independent. Note that (112) and (113) are exact. 

In the approximation error approach, the following Gaussian approximations are 
used: me © N(ex,Te) and tay © N(Ex,u,+V|,). Let the normal approximation 
for the joint density m(e, Ua) be 


T -1 
1 E— &x Tag Ee E— Ex 
1(E, a) & Exp -5( ) ( as ) ( ) (114) 
2 Ma = Ma,* Tyae Tapa Ma —_ Lax 
whence 
Exua = Ex + Tepe P iniig a — Ha) (115) 
Pel, = Dec — Poy, ee Page (116) 


Define the normal random variable v = e + ¢€ so that v| la ~ N(Vejng+P vin): 
Thus, we obtain for the approximate likelihood distribution 
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m(y | ba) & N(y — A(t, bs,0, 0) — Vila? Vy ipa): 


Since we are after computational efficiency, a normal approximation m(q) ~ 
N (fax, ,,) for the prior model is also usually employed. Thus, we obtain the 
approximation for the posterior distribution 


(Maly) & WY | a) We (Ma) (117) 
1 
cc exp (=F Lg 9 — Attar ttces $e) — Malad C18) 


+ |[Lyg (Ha — Ha») II’). (119) 


where!) = oe Lyiu, and 7", = Li, Lua. See [68] or more details on the 
particular problem of marginalizing over fhe: scattering coefficient. 

The approximation error approach has been applied to various kinds of approx- 
imation and modeling errors as well as other inverse problems. Model reduction, 
domain truncation, and unknown anisotropy structures in diffuse optical tomogra- 
phy were treated in [12, 45, 46, 67]. Missing boundary data in the case of image 
processing and geophysical EIT were considered in [23] and [70], respectively. 
Furthermore, in [76-78], the problem of recovery from simultaneous geometry 
errors and model reduction was found to be possible. In [95], the radiative transfer 
model was replaced with the diffusion approximation. It was found that also in this 
kind of a case, the statistical structure of the approximation errors enabled the use 
of a significantly less complex model, again simultaneously with significant model 
reduction for the diffusion approximation. But also here, both the absorption and 
scattering coefficients were estimated simultaneously. 

The approximation error approach was extended to nonstationary inverse prob- 
lems in [51] in which linear nonstationary (heat transfer) problems were considered 
and in [50] and [52] in which nonlinear problems and state space identification 
problems were considered, respectively. A modification in which the approximation 
error statistics can be updated with accumulating information was proposed in [53] 
and an application to hydrogeophysical monitoring in [71]. 


Experimental Results 


In this section, we show an example where the error model is employed for 
compensating the modeling errors caused by reduced discretization accuracy h and 
experimental DOT data is used for the observations. 


Experiment and Measurement Parameters 

The experiment was carried out with the frequency-domain (FD) DOT instrument at 
Helsinki University of Technology [75]. The measurement domain Q is a cylinder 
with radius r = 35mm and height 110mm; see Fig. 7. The target consists of 
homogeneous material with two small cylindrical perturbations, as illustrated in 
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Fig. 7 Top: Measurement 
domain Q. The dots denote 
the location of the sources 
and detectors. The (red) lines 
above and below the sources 
and detectors denote the 
truncated model domain Q. 
The green line denotes the 
central slice of the domain Q. 
Bottom: Central slice of the 
target (Ja left, pu. right) 


Fig. 7. The background optical properties of the phantom are approximately fla he = 
0.01 mm™! and j4s52 = 1mm! at wavelength A ~ 800 nm. The cylindrical 
perturbations, which both have diameter and height of 9.5mm, are located such 
that the central plane of the perturbations coincides with the central xy-plane of 
the cylinder domain Q. For an illustration of the cross sections of j1a and j1{, see 
bottom row in Fig. 7. The optical properties of perturbation | are approximately 
Hap1 = 0.02 mm !, Uspt = Imm! (ie., purely absorption contrast) and the 
properties of perturbation 2 are apo = 0.01 mm !, HMsp2 = 2 mm! (ie., 
purely scatter contrast), respectively. The source and detector configuration in the 
experiment consisted of 16 sources and 16 detectors arranged in interleaved order 
on two rings located 6mm above and below the central xy-plane of the cylinder 
domain. The locations of sources and detectors are shown with dots in Fig. 7. The 
measurements were carried out at A = 785 nm with an optical power of 8 mW and 
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modulation frequency 27 = 100 MHz. The log amplitude and phase shift of the 
transmitted light was recorded at 12 farthermost detector locations for each source, 
leading to a real-valued measurement vector 


_ { re(log(z)) 384 
a a = 


for the experiment. The statistics of measurement noise in the measurement y are 
not known. Thus, we employ the same implicit (ad hoc) noise model that was 
used for reconstructions from the same measurement realization in [88]. The noise 
model is 


e~N(0,T,), 


where I. is a diagonal data scaling matrix which is tuned such that the initial 
(weighted) least squares (LS) residual 


I[Le(y — yo) II? =2, To’ = LILe 


between the measured data y and forward solution yo at the initial guess x = x9 
becomes unity for both data types (log amplitude re(log(z)) and phase im(log(z))). 


Prior Model 

In this study, we use a proper Gaussian smoothness prior as the prior model for the 
unknowns. The absorption and scatter images jz, and j, are modeled as mutually 
independent Gaussian random fields with a joint prior model 


1 . 
(x5) & exp —5llbas (x3 — sso Pt » Lig Tt. (120) 


where 


T — Pua 0 
SLO TJ 


The construction of the blocks T,, and Ty, has been explained for a two- 
dimensional case in [11], and the extension to three-dimensional case is straightfor- 
ward. The parameters in the prior model were selected as follows. The correlation 
length for both jz. and yi in the prior was set as 11 mm. The correlation length 
can be viewed (roughly) as our prior estimate about the expected spatial size of 
perturbations in the target domain. The prior mean for absorption and scatter were 
set as [la, = 0.01 mm! and p., = 1mm !, and the marginal standard deviations 
of absorption and scatter in each voxel were chosen such that 30,, = 0.01 and 
30, = 1, respectively. This choice corresponds to assuming that the values of 
absorption and scatter are expected to lie within the intervals 4, € [0,0.02] and 
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0.001 0.022 0.1 2.2 
0.001 0.022 0.1 2.2 


Fig. 8 Two random samples from the prior density (120). The images display the cross section of 
the 3D parameters at the central slice of the cylinder domain Q2. Left: Absorption 1a. Right: Scatter 


ne 


LL, € [0, 2] with prior probability of 99.7 %. Figure 8 shows two random samples 
from the prior model. 


Selection of FEM Meshes and Discretization Accuracy 

To select the discretization accuracy 6 for the accurate forward model Ag 5(xs), we 
adopted a similar procedure as in [11]. In this process, we computed relative error 
in the FEM solution with respect to the discretization level h and identified 6 as that 
mesh density beyond which the relative error 


|Ag.n — Ag.n|| 
| Ao.n’|| 


in both amplitude and phase parts of the forward solution was stabilized. The mesh 
for the reference model Ag» in the convergence analysis consisted of NV, = 
253,981 node points and Ne = 1,458,000 (approximately) uniform tetrahedral 
elements. We found that the errors in the FEM solution were stabilized when using a 
(uniform) tetrahedral mesh with (approximately) 150,000 nodes or more, and thus, 
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Table 1 Mesh details for test case. N,, is the number of nodes, N, is the number of tetrahedral 
elements in the mesh, and , is the number of voxels in the representation of jz, and 2{. t is the 
wall clock time for one forward solution 


Model Nn Ne Np t (s) 
Ags | 178 
Ag.h 2,413 11,664 7,668 0.4 


0 50 100 150 200 250 300 350 


Fig. 9 Modeling error between the accurate model Ags and target model Ag; see Table 1. 
Left: Covariance structure of the approximation error e. The displayed quantity is the signed 
standard deviation sign(T’,)-/|I'.|, where the product refers to the element-by-element (array) 
multiplication. Right: Normalized eigenvalues A/Amax of T';. of I’; 


we chose for the accurate model Ags a mesh with N, = 148,276 node points. 
For the target model Ag ,, we chose a mesh with N,, = 2,413 nodes; see Table 1. 
For the representation of the unknowns (Ha, Ll), the domain Q was divided into 
np = 7,668 cubic voxels (i.e., number of unknowns n = 15,336) in both models 
Ags and Ag ;,. Thus, the projector P : x3 + x, between the models is the identity 
matrix. 


Construction of Error Models 
To construct the enhanced error model, we proceeded as in section “Approximation 
Error Approach.” The size of the random ensemble S' from the prior model z(x;), 
Eq. 120, was L = 384. Figure 8 shows central xy-slices from two realizations 
of absorption and scatter images from the ensemble (the location of the slice is 
denoted by green line in Fig. 7). Using the ensemble, Gaussian approximations ¢ ~ 
N (e€x, T'-) for the error between the accurate model Ag 5 and the target models were 
computed. 

To assess the magnitude of the modeling error, we estimate signal-to-noise (SNR) 
ratio of the modeling error as 
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SNR = 101o Aa.all? 
£10 \ Teal? + trace(P.) } ’ 


where Ag ; is the mean of the accurate model Ags over the ensemble S$. The SNR 
is estimated separately for the amplitude and phase part of the forward model. 

Consider now the modeling error between the accurate model Ags (N, = 
148,276 nodes) and target model Ag, (N, = 2,413 nodes) in the first test 
case. In this case, the estimated SNRs for the modeling error in log amplitude 
and phase are approximately 20 and 13, corresponding to error levels of 10 and 
22 %, respectively. These error levels exceed clearly typical levels of measurement 
noise in DOT measurements. Left image in Fig.9 displays the covariance matrix 
T’,, revealing the correlation structure of ¢. Combining the high magnitude and 
complicated correlation structure of the modeling error ¢ with the fact that the 
inverse problem is sensitive to modeling errors, one can expect significant artifacts 
in the reconstructions with conventional noise model when employing the target 
model Ag a. 

Right image in Fig.9 shows normalized eigenvalues A/Amax of I’, for the 
modeling error between models Ag 5 and Ag, in the first test case. As can be seen, 
the eigenvalues are decaying rapidly and already the 40th eigenvalue is less than 
1 % of the maximum. Roughly speaking, this rapid decay of the eigenvalues can be 
interpreted such that the variability in the modeling error can be well explained with 
a relatively small number of principal components. In other words, one can take this 
as a sign that the structure of the modeling error is not “heavily dependent” on the 
realization of x or the prior model (x), and thus, the error model can be expected 
to perform well. 

Notice that the setting up of the error model is a computationally intensive 
task, while the use of the model is as with the conventional error model. The 
computation time for setting up the error model is roughly equivalent to the size of 
the ensemble times, the time for forward solution in the accurate and approximate 
models. However, the error model needs to be estimated only once for a fixed 
measurement setup, and this estimation can be offline. 


Computation of the MAP Estimates 

The MAP-CEM and MAP-EEM estimates are computed by a Polak Ribiere conju- 
gate gradient algorithm which is equipped with an explicit line search. Similarly as 
in the initial estimation, the positivity prior of the absorption and scatter images is 
taken into account by using (scaled) logarithmic parameterization 


g , log{ 
Hag Lso 


in the unconstrained optimization process; for details, see [88]. 
Results are shown in Figs. 10 and 11 and computation times in Table 2. 
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0.0073 0.0091 


0.0050 0.0068 0.0076 0.0085 0.61 0.95 1.01 1.12 


Fig. 10 Pure discretization errors. Central horizontal slice from the 3D reconstructions of 
absorption j2, and scattering p14. Top row: MAP estimate with the conventional error model (MAP- 
CEM) using the accurate forward model Ag s (number of nodes in the FEM mesh N,, = 148,276). 
Left: [la.cem. Right: [lazem. Bottom row: MAP estimates with the conventional (MAP-CEM) 
and enhanced error models (MAP-EEM) using the target model Ag, (the number of nodes 
N, = 2,413). Correct model domain Q = Q is used in the target model Ag). Columns from 
left to right: [a,cEM, [/a,EEM> [4s,CEM, and [ls EM. The number of unknowns x = (Ha. 7) T in the 
estimation with both models, Ag.s and Ag, was 15,336 


The images in the top row display the MAP estimate with the conventional noise 
model using the accurate forward model Ags (N, = 148,276). The estimated 
values of global parameters flay = 0.0079mm! and psy = 1.086mm~! are 
relatively close to the background values jtabg = 0.01 mm™! and jusp¢ = 1 mm! 
of the target phantom. As can be seen, the structure of the phantom is reconstructed 
well, but the contrast of the recovered inclusions is low compared to the (presumed) 
contrast. However, the low contrast is related to the measurement setup, not the 
reconstruction algorithm; the same measurement realization has previously been 
used for absolute reconstructions with different algorithm in [88], resulting to 
similar reconstruction quality and contrast in the optical properties. See also [89] 
for similar results with the same measurement system. The MAP-CEM estimate 
with the accurate model Ags can be considered here as a reference estimate 
using conventional noise model in absence of modeling errors caused by reduced 
discretization or domain truncation. 

The MAP-CEM estimate using the coarse target model Ag, (N, = 2,413) 
is shown in the first and third images in the bottom row in Figs. 10 and 11. As 
can be seen, the use of reduced discretization has caused significant errors in the 
reconstruction and also the levels of 44 and pe are erroneous. 
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0.0073 0.0091 


0.0050 0.0068 0.0076 0.0085 0.61 0.95 1.01 1.12 


Fig. 11 Pure discretization errors. Vertical slices from the 3D reconstructions of absorption jl, 
and scattering j4{. The slices have been chosen such that the inclusion in the parameter is visible. 
The arrangement of the images is equivalent to Fig. 10 


Table 2 Reconstruction times for Figs.10 and 11. fini: is the (wall clock) time for initial 
estimation, fap for the MAP estimation, and fo the total reconstruction time (initial + MAP) 


Noise model | Forward model | Finit (s) | tmp (s) trot (S) 

CEM [Ags [126min20s | 173min22s | 299 min 44 
CEM [Ags ‘Imin Ils /7min 18s |8 min 29s 
EEM lay [28s ‘7 min 345 |8min 2s 


The MAP estimate with the enhanced error model using the coarse target model 
Ag 7 is shown in the second and fourth images in the bottom row in Figs. 10 and 11. 
As can be seen, the estimate is very similar to the MAP-CEM estimate with the 
accurate model Ag 3, showing that the use of enhanced error model has efficiently 
compensated for the errors caused by reduced discretization accuracy. These results 
indicate that the enhanced error model allows significant reduction in computation 
time without compromise in the reconstruction quality; whereas the reconstruction 
time for the MAP-CEM using accurate model Ags is very close to 5h, the 
computation time for MAP-EEM is only 8 min. 
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5 Conclusion 


In this chapter, we mainly discussed the use of the diffusion approximation for 
optical tomography. Because of the exponentially ill-posed nature of the corre- 
sponding inverse problem, diffuse optical tomography (DOT) gives low resolution 
images. Current research is focused on several areas: the use of auxiliary (multi- 
modality) information to improve DOT images, the development of smaller-scale 
(mesoscopic) imaging methods based on the radiative transfer equation, and the 
development of fluorescence and bioluminescence imaging techniques which give 
stronger contrast to features of interest. These methods are closely tied to the 
development of new experimental systems and to application areas which are 
driving the continued interest in this technique. 
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Abstract 

Photoacoustic tomography (PAT), also known as thermoacoustic or optoacoustic 
tomography, is a rapidly emerging imaging technique that holds great promise 
for biomedical imaging. PAT is a hybrid imaging technique, and can be viewed 
either as an ultrasound mediated electromagnetic modality or an ultrasound 
modality that exploits electromagnetic-enhanced image contrast. In this chapter, 
we provide a review of the underlying imaging physics and contrast mechanisms 
in PAT. Additionally, the imaging models that relate the measured photoacoustic 
wavefields to the sought-after optical absorption distribution are described in 
their continuous and discrete forms. The basic principles of image reconstruction 
from discrete measurement data are presented, which includes a review of 
methods for modeling the measurement system response. 


1 Introduction 


Photoacoustic tomography (PAT), also known as thermoacoustic or optoacoustic 
tomography, is a rapidly emerging imaging technique that holds great promise for 
biomedical imaging [30, 31,45, 61, 63]. PAT is a hybrid technique that exploits the 
thermoacoustic effect for signal generation. It seeks to combine the high electro- 
magnetic contrast of tissue with the high spatial resolution of ultrasonic methods. 
Accordingly, PAT can be viewed either as an ultrasound mediated electromagnetic 
modality or an ultrasound modality that exploits electromagnetic-enhanced image 
contrast [65]. Since the 1990s, there have been numerous fundamental studies 
of photoacoustic imaging of biological tissue [19, 31, 41, 44, 46, 47,57], and the 
development of PAT continues to progress at a tremendous rate [18, 24, 28, 30, 31, 
33,45, 64, 65]. 

When a short electromagnetic pulse (e.g., microwave or laser) is used to 
irradiate a biological tissue, the thermoacoustic effect results in the emission of 
acoustic signals that can be measured outside the object by use of wideband 
ultrasonic transducers. The objective of PAT is to produce an image that represents 
a map of the spatially variant electromagnetic absorption properties of the tissue, 
from knowledge of the measured acoustic signals. Because the optical absorption 
properties of tissue are highly related to its molecular constitution, PAT images 
can reveal the pathological condition of the tissue [11,26] and therefore facilitate 
a wide range of diagnostic tasks. Moreover, when employed with targeted probes or 
optical contrast agents, PAT has the potential to facilitate high-resolution molecular 
imaging [32,58] of deep structures, which cannot be achieved easily with pure 
optical methods. 

From a physical perspective, the image reconstruction problem in PAT can be 
interpreted as an inverse source problem [6]. Accordingly, PAT is a computed 
imaging modality that utilizes an image reconstruction algorithm to form the 
image of the absorbed optical energy distribution. A variety of analytic image 
reconstruction algorithms have been developed for three-dimensional (3D) PAT, 
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assuming point-like ultrasound transducers with canonical measurement apertures 
(22, 23, 30, 31, 35, 64-66]. All known analytic reconstruction algorithms that are 
mathematically exact and numerically stable require complete knowledge of the 
photoacoustic wavefield on a measurement aperture that either encloses the entire 
object or extends to infinity. In many potential applications of PAT imaging, it 
is not feasible to acquire such measurement data. Because of this, iterative, or 
more generally, optimization-based, reconstruction algorithms for PAT are being 
developed actively [4, 5, 18, 48, 50] that provide the opportunity for accurate 
image reconstruction from incomplete measurement data. Iterative reconstruction 
algorithms also allow for accurate modeling of physical nonidealities in the data, 
such as those introduced by acoustic inhomogeneity and attenuation, or the response 
of the imaging system. 

In this chapter, the physical principles of PAT are reviewed. We start with 
a review of the underlying imaging physics and contrast mechanisms in PAT. 
Subsequently, the imaging models that relate the measured photoacoustic wavefields 
to the sought-after optical absorption distribution are described in their continuous 
and discrete forms. The basic principles of image reconstruction from discrete 
measurement data are presented, which includes a review of methods for modeling 
the measurement system response. We defer a detailed description of analytic 
reconstruction algorithms and the mathematical properties of PAT in » Mathematics 
of Photoacoustic and Thermoacoustic Tomography. 


2 Imaging Physics and Contrast Mechanisms 


In PAT, a laser or microwave source is used to irradiate an object, and the 
thermoacoustic effect results in the generation of a pressure wavefield p(r, t) 
[45, 54,65], where r € R? and ¢ is the temporal coordinate. The resulting pressure 
wavefield can be measured by use of wideband ultrasonic transducers located on a 
measurement aperture 29 C R?, which is a 2D surface that partially or completely 
surrounds the object. In this section, we review the physics that underlies the image 
contrast mechanism in PAT employing laser and microwave sources. 


The Thermoacoustic Effect and Signal Generation 


The generation of photoacoustic wavefields in an inviscid and lossless medium is 
described by the general photoacoustic wave equation [55,56] 


2 2 
(v-s55] Pe ee AD (1) 


Kc? Ot? 


where V? is the 3D Laplacian operator, T(r, 1) denotes the temperature rise within 
the object at location r and time ¢ due to absorption of the probing electromagnetic 
radiation, and p(r, ¢) denotes the resulting induced acoustic pressure. The quantities 
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B, k, and c denote the thermal coefficient of volume expansion, isothermal 
compressibility, and speed of sound, respectively. Because an inviscid medium is 
assumed, the propagation of shear waves is neglected in Eq. (1), which is typically 
reasonable for soft-tissue imaging applications. Note that the spatial-temporal 
samples of p(r,t), which are subsequently degraded by the response of the imaging 
system, represent the measurement data in a PAT experiment. 

When the temporal width of the exciting electromagnetic pulse is sufficiently 
short, the pressure wavefield is produced before significant heat conduction can 
take place. In this situation, the excitation is said to be in thermal confinement. 
Specifically, this occurs when the temporal width t of the exciting electromagnetic 
pulse satisfies [56] 

dz 
T<—, (2) 
Aah 


where d, and @,;, denote the characteristic dimension (m) of the heated region and 
the thermal diffusivity (m*/s). 
Under conditions of thermal confinement, the temperature function 7 (r,t) 
satisfies 
OT (r,t) 
ot 


pCy = HA (r,t), (3) 


where p and Cy denote the mass density (kg/ nm) and specific heat capacity of the 
medium at constant volume. The quantity H(r,t) [J/(m*s)] is called the heating 
function that describes the energy per unit volume and time that is deposited in the 
medium by the exciting electromagnetic pulse. On substitution from Eq. (3) into 
Eq. (1), one obtains the simplified photoacoustic wave equation 


1 F) __ B dH(r,1) 
(v Pe.) =-— a (4) 


where C, = pc*k Cy [J/(kg K)] denotes the specific heat capacity of the medium at 
constant pressure. It is sometimes convenient to work the velocity potential ¢(r, t) 
that is related to the pressure as p(r,t) = —pieen It can be readily verified that 
Eq. (4) can be reexpressed in terms of (r,t) as 


2 
(v = 33) o(r,t) = FH). (5) 
P 


c2 Or 


The photoacoustic wave equations described by Eqs. (4) and (5) have been 
solved for a variety of canonical absorbers [15-17]. Figure 1 shows an example 
corresponding to a uniform spherical absorber. In this case, the optical absorber 
was assumed to possess a speed of sound co that matched the background medium. 
Note that the pressure possesses an ““N-shape” waveform. Solutions have also been 
derived for the case where the optical absorbers have acoustical properties that are 
different from those of the background medium [15]. 
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Fig. 1 The pressure (a) and Pressure at r= 2 RF, 
velocity potential (b) 
waveforms produced by the 
thermoacoustic effect for a 
uniform sphere of radius R; 
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In practice, it is appropriate to consider the following separable form for the 
heating function 


A(r,t) = A(r)I(¢), (6) 


where A(r) (J/m?) is the absorbed energy density and /(t) denotes the temporal 
profile of the illuminating pulse. 


When the exciting electromagnetic pulse duration t is short enough to satisfy the 
acoustic stress-confinement condition 


T<—, (7) 
Cc 


in addition to the thermal-confinement condition in Eq. (2), one can approximate 
I(t) by a Dirac delta function /(t) ~ 6(t). Physically, Eq. (7) requires that all of 
the thermal energy has been deposited by the electromagnetic pulse before the mass 
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density or volume of the medium has had time to change. In this case, the absorbed 
energy density A(r) is related to the induced pressure wavefield p(r,t) att = 0 as 


pir.t =0) =TAGQ), (8) 


where I" is the dimensionless Gruneisen parameter. As discussed in detail later, the 
goal of PAT is to determine A(r) or, equivalently, p(r, t = 0) from measurements of 
p(r,t) acquired on a measurement aperture. It is also useful to note that under the 
acoustic stress-confinement condition, Eq. (4) coupled with appropriate boundary 
conditions is mathematically equivalent to the initial value problem [34] 


pile (r,t) =0 (9) 
oie 
subject to 
pi.t=0=T4@) and PED) X (10) 
ot i=0 


The effects of heterogeneous speed of sound or acoustic attenuation are not 
addressed above, but will be described later. In the following two subsections, a 
review of the physical object properties that give rise to image contrast, that is, 
variations in A(r), are reviewed for the case of optical and microwave illumination. 


Image Contrast in Laser-Based PAT 


When an optical laser pulse is employed to induce the thermoacoustic effect, the 
heating function can be explicitly expressed as 


H(r,t) = a(t) ®(r, 1), (11) 


where [q(r) (1/m) is the optical absorption coefficient of the medium and 
®(r, t) [J/(m7s)] is the optical fluence rate [39]. Assuming ®(r,t) = ®,(r)/(1), 
Eq. (11) can be expressed as 
A(r,t) = fat) ®s(r) L(t), (12) 
—-— 
A(r) 


where the absorbed energy density, which is the sought-after quantity in PAT, is now 
identified as 


A(r) = La(r) ®;(r). (13) 


Equation (13) reveals that image contrast in laser-based PAT is determined by the 
optical absorption properties of the object as well as variations in the fluence of 
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the illuminating optical radiation. Because only the optical absorption properties 
are intrinsic to the object, in implementation of PAT it is desirable to make the 
optical fluence ®,(r) as uniform as possible so one can unambiguously interpret 
A(r) & [g(r). This presents experimental challenges, and computational methods 
for quantitative determination of j,(r) are being developed actively [9, 12, 68]. 
However, in most current implementations of PAT, an estimate of A(r) represents 
the final image. 

There are many desirable characteristics of laser-based PAT for biological 
imaging. The optical absorption coefficient jz,(r) is a function of the molecular 
composition of tissue [11] and is therefore sensitive to tissue pathologies and 
functions. Specifically, PAT can deduce physiological parameters such as the 
oxygen saturation of hemoglobin and the total concentration of hemoglobin, as 
well as certain features of cancer such as elevated blood content of tissue due to 
angiogenesis [65]. 

Although pure optical imaging methods also are sensitive to such physiological 
parameters, they are limited by their relatively poor spatial resolution and inability 
to image deep tissue structures. PAT circumvents these limitations because diffusely 
scattered photons that are absorbed at deep locations are still useful for signal 
generation via the thermoacoustic effect. When the wavelength of the optical source 
lies in the range 700-900nm, light can penetrate up to several centimeters in 
biological tissue. As described by Eq. (13), the optical fluence ®,;(r), which contains 
ballistic and diffusely scattered photons, modulates j,(r). However, as described 
later, the spatial resolution of the reconstructed estimate of A(r) is not directly 
affected by this and is determined largely by the properties of the measured pressure 
signal p(r, ft). 


Image Contrast in RF-Based PAT 


When an RF pulse is employed to induce the thermoacoustic effect, the nature of 
the image contrast is different from that described above. A detailed analysis of this 
has been conducted by Li et al., in [40]. Consider the case of an RF pulse whose 
temporal width is much longer than the oscillation period of the electromagnetic 
wave at the center frequency w,. The RF source is assumed to produce a plane- 
wave with linear polarization and can be described as 


Cin(t) = S(t)cos(@-t), (14) 


where S(t) is a slowly varying envelope function. Furthermore, consider that the 
medium is isotropic and the electrical conductivity of the medium o(r, w) can be 
approximated as 


o(r,@) © a(¥, a), (15) 


where w represents the temporal frequency variable. Under the stated conditions, it 
is the short-time averaged heating function 
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< H@,t)>= val " IH(e.D), (16) 
where T, = =, which gives rise to signal generation in RF-based PAT [40]. It has 
been demousiated [40] that this quantity can be expressed as 

< H(,t) >= AS, (17) 
where © so ) represents the electric field intensity of the RF source and 
Agy = CE MDEO. oP ae 


|@in(@e)|? 


where E(r, w,) and éin(w,) denote the temporal Fourier transforms of E(r, ft) and 
€in(t) evaluated at @ = @,, with E(r,t) denoting the local electric field. Note that 
Eq. (18) represents the quantity that is estimated by conventional PAT reconstruction 
algorithms. 

Equation (18) reveals that image contrast in RF-based PAT is determined by 
the electrical conductivity of the material, which is described by the complex 
permittivity, as well as variations in the illuminating electric field at temporal 
frequency component w = w,. Because only the electrical conductivity is intrinsic 
to the object material, it is desirable to make |E (r, @¢)|? as uniform as possible, 
so one can unambiguously interpret as the distribution of the conductivity. It 
has been demonstrated in computer-simulation and experimental studies [40] that 
estimates of A(r) produced by conventional image reconstruction algorithms can be 
nonuniform and contain distortions due to diffraction of the electromagnetic wave 
within the object to be imaged. There remains a need to develop improved image 
reconstruction methods to mitigate these. 

The complex permittivity of tissue has a strong dependence on the water content, 
temperature, and ion concentration. Because of this, any variations in blood flow in 
tissue will give rise to changes in the quantity of water and consequently to changes 
in its complex permittivity. RF-based PAT therefore has the high sensitivity to tissue 
properties of a microwave technique but requires solution of a tractable acoustic 
inverse source problem for image reconstruction. 


Functional PAT 


A highly desirable characteristic of PAT is its ability to provide detailed functional, 
in additional to anatomical, information regarding biological systems. In this 
section, we provide a brief review of functional imaging using PAT. For additional 
details, the reader is referred to parts IX and X in reference [55] and the references 
therein. 
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Due to optical contrast mechanism discussed in section “Image Contrast in 
Laser-Based PAT,” laser-based functional PAT operating in the near-infrared (NIR) 
frequency range can be employed to determine information regarding the oxy- 
genated and deoxygenated hemoglobin within the blood of tissues. This can permit 
the study of vascularization and hemodynamics, which is relevant to brain imaging 
and cancer detection. 

Functional PAT imaging of hemoglobin can be achieved by exploiting the 
known characteristic absorption spectra of oxygenated hemoglobin (HbO2) and 
deoxygenated hemoglobin (Hb). Consider the situation where the optical fluence 
®,(r) is known, and therefore the optical absorption coefficient ,(r) can be 
determined from the reconstructed absorbed energy density A(r) via Eq. (13). Let 
pi (r) and x2 (r) denote the reconstructed estimates of ju, (r) corresponding to the 
cases where the wavelength of the optical source is set at A; and A2. From knowledge 
of these two estimates, the hemoglobin oxygen saturation distribution, denoted by 
SO2(r), is determined as 


Xd x 
pe (req — pe (r)enp 


SO2(r) = ; 
1a! (Keay — Ha? (EA 


(19) 


where eh and tivo, denote molar extinction coefficients of Hb and HbO>, and 


nee = Hibs - eh The distribution of the total hemoglobin concentration, denoted 
by HbT(r), can be determined as 


a A a A 
Ma" (Ke AH — ba (KE AnD 
hi As : 


HbT(r) = a 
ue — 6 en, 
Hb€HbO) — €Hb€HbOD 


(20) 


An experimental investigation of functional PAT imaging of a rat brain was 
described in [59]. While in different physiological states, a rat was imaged using 
laser light at wavelengths 584 and 600 nm to excite the photoacoustic signals. A two- 
dimensional (2D) scanning geometry was employed, and the estimates of A(1r) were 
reconstructed by use of a backprojection reconstruction algorithm. Subsequently, 
estimates of SO2(r) and HbT(r) were computed and are displayed in Fig. 2. 


3 Principles of PAT Image Reconstruction 


In the remainder of this chapter, we describe some basic principles that underlie 
image reconstruction in PAT. We begin by considering the image reconstruc- 
tion problem in its continuous form. Subsequently, issues related to discrete 
imaging models that are employed in iterative image reconstruction methods are 
reviewed. 

A schematic of a general PAT imaging geometry is shown in Fig.3. A short 
laser or RF pulse is employed to irradiate an object and, as described earlier, the 
thermoacoustic effect results in the generation of a pressure wavefield p(r, t). The 
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normoxia; 600 nm wavelength 
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Fig. 2. Noninvasive spectroscopic photoacoustic imaging of HbT and SOz in the cerebral cortex 
of a rat brain. (a) and (b) Brain images generated by 584 and 600 nm laser light, respectively; (c) 
and (d) image of SO, and HbT in the areas of the cortical venous vessels (Reproduced from Wang 
et al. [59]) 


pressure wavefield propagates out of the object and is measured by use of wideband 
ultrasonic transducers located on a measurement aperture Qo9 C R?, which is 
a 2D surface that partially or completely surrounds the object. The coordinate 
ro € Qo will denote a particular transducer location. Although we will assume 
that the ultrasound transducers are point-like, it should be noted that alternative 
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Fig. 3 A schematic of the PAT imaging geometry 


implementations of PAT are being actively developed that employ integrating 
ultrasound detectors [24, 49]. 


PAT Imaging Models in Their Continuous Forms 


When the object possesses homogeneous acoustic properties that match a uniform 
and lossless background medium, and the duration of the irradiating optical pulse is 
negligible (acoustic stress confinement is obtained), the pressure wavefield p(ro, t) 
recorded at transducer location rg can be expressed [65] as a solution to Eq. (9): 


|ro—rl 
d é(t— ey 
eS Pe fyngt ed (21) 


where co is the (constant) speed of sound in the object and background medium. 
The function A(r) is compactly supported, bounded, and nonnegative, and the inte- 
gration in Eq. (21) is performed over the object’s support volume V.. Equation (21) 
represents a canonical imaging model for PAT. The inverse problem in PAT is to 
determine an estimate of A(r) from knowledge of the measured p(ro,t). Note 
that, as described later, the measured p(ro,t) will generally need to be corrected 
for degradation caused by the temporal and spatial response of the ultrasound 
transducer. 

The imaging model in Eq. (21) can be expressed in an alternate but mathemati- 
cally equivalent form as 


g(ro.t) = far a(n) 8 (1 mat), (22) 
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where the integrated data function g(ro, ¢) is defined as 


4nC 
B 


t 
g(ro,t) = Pr i dt’ p(to.t’). (23) 
0 


Note that g(ro,f) represents a scaled version of the acoustic velocity potential 
(Yo, t). Equation (22) represents a spherical Radon transform [22,43] and indicates 
that the integrated data function describes integrals over concentric spherical 
surfaces of radii cot that are centered at the receiving transducer location ro. When 
these spherical surfaces can be approximated as planes, which would occur when 
imaging sufficiently small objects that are placed at the center of the scanning 
system, Eq. (22) can be approximated as a 3D Radon transform [30, 31]. 


Universal Backprojection Algorithm 


A number of analytic image reconstruction algorithms [22, 34, 64, 65] for PAT have 
been developed in recent years for inversion of Eq. (21) or (22). A detailed descrip- 
tion of analytic algorithms will be provided in » Mathematics of Photoacoustic 
and Thermoacoustic Tomography. However, the so-called universal backprojection 
algorithm [64] is reviewed below. 

The three canonical measurement geometries in PAT employ measurement 
apertures (29 that are planar [66], cylindrical [67], or spherical [61]. The universal 
backprojection algorithm proposed by Xu and Wang [64] has been explicitly derived 
for these geometries. In order to present the algorithm in a general form, let S denote 
a surface, where S = Qo for the spherical and cylindrical geometries. For the planar 
geometry, let S = Qo + QQ; where Q6 is a planar surface that is parallel to Qo and 
the object resides between (29 and (25. 

It has been verified that the initial pressure distribution p(r, t = 0) = T'A(r) can 
be mathematically determined from knowledge of the measured p(fo, ft), ro € Qo, 
by use of the formula 


1 oe is 
p(r,t =0) = > d as [ dk p(ro,k) [ni : Woe Cr, ro) (24) 
S —oo 


where p(ro,k) denotes the temporal Fourier transform of p(ro,f) that is defined 
with respect to the reduced variable f = cot as 


P(to, k) = fa P(¥o,t) exp(ikt). (25) 


Here, n> denotes the unit vector normal to the surface S pointing toward the source, 


Vo denotes the gradient operator acting on the variable ro, and Gir, Yo) = 
exp(—ik|r—rol) 


Z is a Green’s function of the Helmholtz equation. 
a|r—ro| 
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Equation (24) can be expressed in the form of a filtered backprojection algo- 
rithm as 


= |r—Yrol) 
’ 


Sp (26) 


7 
pint =0)= | 3 Me 
x0 


where %o is the solid angle of the whole measurement surface (29 with respect to the 
reconstruction point inside (29. Note that X9 = 47 for the spherical and cylindrical 
geometries, while X& 9 = 2z for the planar geometry. The solid angle differential 
d&Xo is given by 


do = dQo n> -(r—ro) 


= 27 
Fon? [r—rol ae 


where d Q2o is the differential surface area element on Qo. The filtered data function 
b(ro, f) is related to the measured pressure data as 


b(r,2) = 2p(ro,?) — 27 (28) 


-dp(¥o, f) 
ae 
Equation (26) has a simple interpretation. It states that p(r,t = 0), or 
equivalently A(r), can be determined by backprojecting the filtered data function 
onto a collection of concentric spherical surfaces that are centered at each transducer 
location ro. 


The Fourier-Shell Identity 


Certain insights regarding the spatial resolution of images reconstructed in PAT can 
be gained by formulating a Fourier domain mapping between the measured pressure 
data and the Fourier components of A(r) [6]. Below we review a mathematical 
relationship between the pressure wavefield data function and its normal derivative 
measured on an arbitrary aperture that encloses the object and the 3D Fourier 
transform of the optical absorption distribution evaluated on concentric (Ewald) 
spheres [6]. We have referred to this relationship as a “Fourier-shell identity,’ which 
is analogous to the well-known Fourier slice theorem of X-ray tomography. 

Consider a measurement aperture (29 that is smooth and closed, but is otherwise 
arbitrary, and let § € S* denote a unit vector on the 3D unit sphere S*. The 3D spatial 
Fourier transform of A(r), denoted as A(v), is defined as 


A(v) = [arac) eur, (29) 
Vv 


where the 3D spatial frequency vector v = (vx, vy, vz) is the Fourier conjugate of r. 
It has been demonstrated [6] that 
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iC» 


A(v = ks) = isi 


S' [i -VBp (06k) + ika’ <8 p (15, k) Je, 


(30) 
where p(ro, k) is defined in Eq. (25), dS’ is the differential surface element on Qo, 
and ni’ is the unit outward normal vector to Qo at the point Yi € Qo. Equation (30) 
has been referred to as the Fourier-shell identity of PAT. Because § can be chosen 
to specify any direction, A(v = kS) specifies the Fourier components of A(r) 
that reside on a spherical surface of radius |k|, whose center is at the origin. 
Therefore, Eq. (30) specifies concentric “shells” of Fourier components of A(r) 
from knowledge of p(ro, k) and its derivative along the 7’ direction at each point on 
the measurement aperture. As reviewed below, this will permit a direct and simple 
analysis of certain spatial resolution characteristics of PAT. 

For a 3D time-harmonic inverse source problem, it is well known [10, 13, 14] that 
measurements of the radiated wavefield and its normal derivative on a surface that 
encloses the source specify the Fourier components of the source function that reside 
on an Ewald sphere of radius k = 2, where w is the temporal frequency. In PAT, 
the temporal dependence /(t) of the heating function H (r,t) is not harmonic and, 
in general, /(k) 4 0. In the ideal case where /(t) = 8(t), [(k) = c. Consequently, 
when Eq. (30) is applied to each temporal frequency component k of p(ro,<), 
the entire 3D Fourier domain, with exception of the origin, is determined by the 
resulting collection of concentric spherical shells. This is possible because of the 
separable form of the heating function in Eq. (6). 


Special Case: Planar Measurement Geometry 

The Fourier-shell identity can be used to obtain reconstruction formulas for 
canonical measurement geometries. For example, consider the case of an infinite 
planar aperture (9. Specifically, we assume a 3D object is centered at the origin of 
a Cartesian coordinate system, and the measurement aperture QQ 9 coincides with 
the plane y = d > R, where R is the radius of the object. In this situation, 
ry, = (x',d,z’), dS’ = dx'dz, and nr’ = ¥, where denotes the unit vector 
along the positive y-axis. The components of the unit vector § will be denoted as 
(Sx, Sy, Sz). Equation (30) can be expressed as the following two terms: 


A(v = k8) = Ai(v = k8) + An(v = kS), (31) 
where 
Aw=ko= _1Cp  -ikasy If. dx'd? Per yet) a zk) erik (sset2!89) 
k a (k) ay 
(32) 
and 
Ax(v =ks)= Cp sy e ikdsy / dx'dz B(x’, d,z,k) ek! se t2's2) (33) 
cBI(k) oo 
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where, without confusion, we employ the notation p(x, y,z,k) = p(t, k). 
It can be readily verified that Eqs. (32) and (33) can be reexpressed as 


~ iC ; 0 - 
Aj\(v=ks)= Feo thay D(ksy, y, Ks-,k 34 
1( ) kepi (k) ay Pt Sx V,KSz Mines (34) 
and 
A,(v =k) = a e tkdsy B(ks,,d,ksz,k), (35) 
cB1(k) . 


where p(v,, y, vz, k) is the 2D spatial Fourier transform of p(x, y, z, k) with respect 
to x and z (the detector plane coordinates): 


- 1 ‘ 
P(x, ¥,¥2,k) = — [faxa: B(x, y,z, ky ets te), (36) 
42? J Joo 


The free-space propagator for time-harmonic homogeneous wavefields (see, e.g., 
Ref. [36], Chapter 4.2) can be utilized to compute the derivative in Eq. (34) as 


Op(ksy, y, ksz,k = 
OP{Ksx, ¥» bse, k) = ik,/1—s2 —s? p(ks,, y,ksz,k) =iksy p(ks,, y,ks,,k), 


dy 
(37) 
where s, > 0. Equations (34)—(37) and (31) establish that 
A(v = k8) =2A.(0v=kS) for sy > 0. (38) 
Equation (38) permits estimation of A(v = kS) on concentric half shells in the 


domain v, > O and is mathematically equivalent to previously studied Fourier- 
based reconstruction formulas [29, 66]. Note that A(r) is real valued, and therefore 
the Fourier components in the domain v, < 0 can be determined by use of the 
Hermitian conjugate symmetry property of the Fourier transform. 


Spatial Resolution from a Fourier Perspective 


The Fourier-shell identity described in section “The Fourier-Shell Identity” is a 
convenient tool for understanding the spatial resolution characteristics of PAT. 
Below, we analyze the effects of finite transducer temporal bandwidth and aperture 
size on spatial resolution [6, 62]. The analysis is applicable to any measurement 
aperture 29 that corresponds to a coordinate surface of a curvilinear coordinate 
system. 
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Effects of Finite Transducer Bandwidth 

Consider a point-like ultrasonic transducer whose temporal filtering characteristics 
are described by the transfer function B(k:r9). The ro-dependence of B(k:¥0) 
permits transducers located at different measurement locations to be characterized 
by distinct transfer functions. The temporal Fourier transform of the measured 
pressure signal that has been degraded by the temporal response of the transducer 
will be denoted as (ro, x), in order to distinguish it from the ideal pressure signal 
P(to, kK). Because the temporal transducer response can be described by a linear 
time-invariant system, the degraded and ideal pressure data are related as 


Po(to.k) = B(k:¥0) p(to. k). (39) 


Consider the case where (29 corresponds to a coordinate surface of a curvilinear 
coordinate system, that is, ro € 2p is a vector that varies in only two of its 
three components. For such surfaces, B (k; r) can be interpreted as a 3D function 
that does not vary in the vi’ direction and therefore n’- VB (k: r) = 0. If the 
Fourier-shell identity in Eq. (30) is applied with the degraded data function pp (To, k) 
replacing the ideal data, the 3D Fourier components of the resulting image, denoted 
by A;,(r), are recovered as 


= iC ss sprct 
Ap(v = kS) = ats dS' B (k:r)) [a’- Vp (rh, k) + iki’ r), k)jei*70, 
’ kKBI(k) Jao (kK; 10) [ P (tk) 5p (19, k)] ne 


On comparison of Eqs. (30) and (40), we observe that the spatially variant transducer 
transfer function B(k; 19) modulates the integrand of the Fourier-shell identity. In 
this general case, the spatial resolution of A(r) will be spatially variant. 

If a collection of identical transducers spans Qo, B(k;r0) = B(k) will not 
depend on ro and Eq. (40) reduces to the simple form 


A,(v = k$) = B(kK)A(v = ks), (41) 


where A(v = kS) is the exact Fourier data as defined in Eq. (30). As shown in 
Fig. 4, the one-dimensional (1D) transfer function B(k) of the transducer serves as 
a radially symmetric 3D filter that modifies A(v = kS). This establishes that the 
image degradation is described by a shift-invariant linear system: 


Ap(r) = A(r) * B(r), (42) 


where * denotes a 3D convolution and 


Br) = B((rl) = he dk 8) (43) 


is the point-spread function. Equation (43) is consistent with the results derived in 
Ref. [62]. 
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O| k_low k_high 


Fig. 4 (a) An example of a transducer transfer function B(k). (b) The 1D function B(k) acts as 
a radially symmetric filter in the 3D Fourier domain. The shaded region indicates the bandpass 
of 3D Fourier components that results from application of Eq. (41) (Reproduced from Anastasio 
et al. [6]) 


Effects of Nonpoint-Like Transducers 

In addition to a nonideal temporal response, a transducer cannot be exactly point- 
like and will have a finite aperture size. To understand the effects of this on spatial 
resolution, we consider here a transducer that has an ideal temporal response (i.e., 
Bik; ro) = 1) but a finite aperture size [6]. We will assume that the surface of the 
transducer aperture is a subset of the measurement aperture Qo. 

It will be useful to employ a local 3D coordinate system whose origin coincides 
with the center of the detecting surface 7 C Qo, for a transducer at some arbitrary 
but fixed location Yo € Qo. A vector in this system will be denoted as r;, and the 
collection of rz € Qy, spans all locations on the detecting surface of this transducer. 
For a transducer located at a different position rg € Qo, the local coordinate vector 
will be denoted as 


r= Te iril, (44) 
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where 7T,,{-} denotes the corresponding coordinate transformation. This indicates 
that the collection of vectors r corresponding to r, € Qz, reside in a local 
coordinate system whose origin is at rg and span all locations on the detecting 
surface of the transducer centered at that location. 

The measured pressure data p, (ro, k), where the subscript “a” denotes the data 


are obtained in the presence of a finite aperture, can be expressed as 
Baltonk) = f dS, WOrL)B (to+4f-K). (45) 
QL 


where dS, is the differential surface element on QQ, and the aperture function 
W(r_) describe the sensitivity of the transducers at location rz, on their surfaces. We 
assume the aperture function is identical for all transducers, and therefore W(r,) can 
be described simply in terms of the local coordinate r;,. Note that r 4 is a function 
of rz, as described by Eq. (44). 

If the Fourier-shell identity in Eq. (30) is applied with the degraded data function 
Pa(to,k), the 3D Fourier components of the corresponding image A,(r) are 
recovered as 


iCy 


Ag(v = k8) = dS, W 
(v =ks) ii Jo; Si Wrz) 
dQ eS [Al Vp (th trp. k) + ika’-s p (rp +4rP,k)]. 
Qo 


(46) 


By use of the change-of-variable rp = rj) +r ” in Eq. (46), one obtains 


: iCy 
ks ~ KBI(k) fae Lt W(rL) 


dQ eT) Tf! VB (ro, k) + ikA!-5 Bro, k)], (47) 
Qo 


which cannot be simplified further. 

The fact that Eq. (47) does not reduce to a simple form analogous to Eq. (41) 
reflects that the image degradation due to a finite transducer aperture is generally not 
described by a shift-invariant system [62]. A shift-invariant description is obtained 
for planar apertures where Eq. (44) reduces to r 2 = ry, wherer : no longer has a 
dependence on ro. In this case, Eq. (47) can be expressed as 


Ag(v = k8) = W(kS)Ag(v = ks), (48) 


where 


Wks) = / dS; W(r,) el, (49) 
Qr 
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Because, in this case, rz resides on a plane and W(r_) is a real-valued function, 
Eq. (49) corresponds to the complex conjugate of the 2D Fourier transform of the 
aperture function. The point-spread function obtained by computing the 3D inverse 
Fourier transform of Wks ) reduces to a result given in [62]. 


4 Speed-of-Sound Heterogeneities and Acoustic 
Attenuation 


In practice, the object to be imaged may not possess uniform acoustic properties, 
and the images reconstructed by use of algorithms that ignore this can contain 
artifacts and distortions. Below, we review some methods that can compensate for 
an object’s frequency-dependent acoustic attenuation and heterogeneous speed-of- 
sound distribution. 


Frequency-Dependent Acoustic Attenuation 


Because the thermoacoustically inducted pressure signals measured in PAT are 
broadband and ultrasonic attenuation is frequency dependent, in certain applications 
it may be important to compensate for this effect. Below, we describe a method 
described in [37] for achieving this. 

Acoustic waves propagating in a lossy medium are attenuated with a linear 
attenuation coefficient w(@) of the general form [53] 


a(o) = a |o|", (50) 


where w is the angular frequency of the wave [53]. For ultrasonic waves in tissue, 
n= Landay © (1077/2z) cm™! rad“! s. 

Assuming a uniform speed-of-sound distribution, a photoacoustic wave equation 
with consideration of acoustic attenuation can be expressed as [37] 


1 0? B 0 
V-p(r,t) - =—<p (rt L(t) * p (r,t) = -——A(r)—I (ft), 51 
PUR saaP N+ LO * PE) =-ZAS TO. 6D 
where * denotes temporal convolution, co is now a reference phase velocity, and the 
function L(t) describes the effect of acoustic attenuation and is defined as 


i Gale 5 wo 
Lit) = al dw (Kw - =) exp(—ia@t), (52) 
27 Joo oh 
where 
K(w) = + ia(o). (53) 
c(@) 
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Note that in this section, p (r,t) denotes the pressure that is affected by acoustic 
attenuation. 

The phase velocity, denoted here by c(w), also has a temporal frequency 
dependence according to the Kramers—Kronig relations. For = 1, this relationship 
is given by 


1 1 2 
= ———agin (54) 
c(@) co 0 
where is the reference frequency for which c(@9) = Co. 
Let 7 (r, w) denote the temporal Fourier transform of the pressure data: 

CO 
P(r,@) = / dt p(r,t)exp(iwt). (55) 

—0o 


It has been shown [37] that the Fourier transform of the attenuated data D (r, w) 
is related to the unattenuated data p jgeqi (1, t) as 


-1 
P (r,w) = I(@) (3 a jcoaoseno)) 


(o) 
x | Pica (r.tyexp fi Jot + seas [ol | dt, (56) 


where 


‘| 


Pideal (8,1) = it c, | ae'aee) : a (-"") 


dt 4x|r—r’| ’ 


(57) 


is the solution to the photoacoustic wave equation in the absence of attenuation. 

Equation (56) permits one to investigate the effect of acoustic attenuation in PAT. 
It can also be discretized to produce a linear system of equations that can be inverted 
numerically for removal of the effects of acoustic dispersion in the measured 
photoacoustic signals. Subsequently, a conventional PAT image reconstruction 
algorithm could be employed to estimate A(r). A numerical example of this is 
provided in Ref. [37]. 


Weak Variations in the Speed-of-Sound Distribution 
The conventional PAT imaging models described in section “PAT Imaging Models 


in Their Continuous Forms” assume that the object’s speed of sound is constant and 
equal to that of the background medium. In certain biomedical imaging applications, 
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this assumption does not reasonably hold true. For example, the speed of sound 
of breast tissue can vary from 1,400 to 1,540 m/s. Acoustic inhomogeneities can 
introduce significant wavefront aberrations in the photoacoustic signal that are not 
accounted for in the available reconstruction algorithms. 

For a weakly acoustic scattering object, with consideration of phase aberrations 
due to the acoustic heterogeneities effects, the forward PAT imaging model can be 
expressed as a generalized Radon transform [4, 63] 


cot ¢ (1, ro) 


; (58) 
[ro —r| 


&(10,f) = [ae A(r)d[f — cots (r, ro)] 


where f(r, ro) is the time of flight (TOF) for a pressure wave to travel from point 
r within the object to transducer location rp. For objects possessing weak acoustic 
heterogeneities, the TOF can be computed accurately as 


; 1 
iver) = | d°*r oe’ (59) 


’EL(r,ro) 


where c(r) is the spatially variant acoustic speed and the set L(r,ro) describes a 
line connecting ro and r. 

The generalized Radon transform describes weighted integrals of A(r) over iso- 
TOF surfaces that are not spherical in general. The iso-TOF surfaces are determined 
by the heterogeneous acoustic speed distribution c(r) of the object. In the absence 
of acoustic heterogeneities, these are spherical surfaces with varying radii that are 
centered at rp, and Eq. (58) reduces to the spherical Radon transform in Eq. (22). To 
establish this imaging model for PAT imaging, the iso-TOF surfaces that Eq. (58) 
integrates over need to be determined explicitly by use of a priori knowledge of the 
speed-of-sound distribution c(r). Estimates of c(r) can be obtained by performing 
an adjunct ultrasound computed tomography study of the object [25]. Subsequently, 
ray-tracing methods can be employed to identify the iso-TOF surfaces for each 
transducer position rp. Once these path lengths are computed, the points in the object 
that have the same path lengths can be grouped together to form iso-TOF surfaces. 
No known analytic methods are available for inversion of Eq. (58). Accordingly, 
iterative methods have been employed for image reconstruction [3, 25]. 

A higher-order geometrical acoustics-based imaging model has also been 
recently proposed [42] that takes into account the first-order effect in the 
amplitude of the measured signal and higher-order perturbation to the travel 
times. By incorporating higher-order approximations to the travel time that 
incorporates the effect of ray bending, the accuracy of reconstructed images was 
significantly improved. More general reconstruction methods based on the concept 
of time-reversal are discussed in chapter » Mathematics of Photoacoustic and 
Thermoacoustic Tomography. 
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5 Data Redundancies and the Half-Time Reconstruction 
Problem 


In this section, we review data redundancies that result from symmetries in the 
PAT imaging model [2, 5,51, 70], which are related to the so-called half-time 
reconstruction problem of PAT [4]. Specifically, we describe how an image can 
be reconstructed accurately from knowledge of half of the temporal components 
recorded at all transducer locations on a closed measurement aperture. 


Data Redundancies 


Consider the spherical Radon transform imaging model in Eq. (22). Two half-time 
data functions g“ (ro, 7) and g) (rg, f) can be defined as 


On. A= g(to.t) : Ro—Ra St < Ro 
g (0,2) : otherwise, (60) 
and 
z Yo.f) : Ro<f<Rot+R 
g(r9,7) = }OOM een (61) 
0 otherwise. 


Here, Ro denotes the radius of the measurement aperture Qo, and Ry denotes the 
radius of support of A(r). We assume that the object is acoustically homogeneous 
with speed of sound cy and = cyt. Note that the data functions g“ (ro, 7) and 
g (ro,f) each cover different halves of the complete data domain Qo x [Ro — 
R4, Ro + Ra], and therefore g(ro,f) = g (10, 2) + g(r, 7). 

In the limit where Ro — co, the spherical Radon transform reduces to a 
conventional Radon transform that integrates over 2D planes. In that case, an 
obvious conjugate-view symmetry exists [10], and therefore, either of the half- 
time data functions g“(r9,7) and g®(ro,7) contains enough information, in a 
mathematical sense, for exact image reconstruction. Accordingly, a twofold data 
redundancy exists because the complete data function g(ro,7) contains twice as 
much information as is theoretically necessary for exact image reconstruction. 

In the case where Ro is finite, a simple conjugate view symmetry does not exist. 
Nevertheless, it has been demonstrated that a twofold data redundancy exists in the 
complete data function g(ro, f). This has been heuristically [50] and mathematically 
[5] by use of a layer-stripping procedure [2,5,50,70]. This established that A(r) can 
be recovered uniquely and stably from knowledge of either half-time data functions 
g(r, 2) or g(r, f). A similar conclusion has been derived in Ref. [22] using a 
different mathematical approach. 

Analytic inversion formulae for recovering A(r) from knowledge of the half-time 
data functions g(r, #) or g(r, f) are not currently available. However, iterative 
reconstruction algorithms can be employed [4,5] to determine A(r). 
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Mitigation of Image Artifacts Due to Acoustic Heterogeneities 


If the spatially variant speed of sound c(r) is known, one can numerically invert a 
discretized version of Eq. (58) to determine an estimate of A(r) [3]. However, in 
many applications of PAT, c(r) is not known, and images are simply reconstructed 
by use of algorithms that assume a constant speed of sound. This can result in 
conspicuous image artifacts. 

Let £(ro,¢) denote a data function that is contaminated by the effects of speed- 
of-sound variations within the object that is related to A(r) according to Eq. (58). 
Let A(r) denote an estimate of A(r) that is reconstructed from &(ro,7) by use of 
a conventional reconstruction algorithm that assumes an acoustically homogeneous 
object. The quantities ¢" (ro, 7) and ¢ (ro, f) denote half-time data functions that 
are defined in analogy with Eqs. (60) and (61) with g(ro, 7) replaced by (ro, f). 

An image reconstructed from g“) (ro, 7) can sometimes contain reduced artifact 
levels as compared to one reconstructed from the complete data (ro,f). To 
demonstrate this, in the discussion below, we consider the 2D problem and the 
spatially variant speed-of-sound distribution shown in Fig. 5. This speed-of-sound 
distribution is comprised of two uniform concentric disks that have cp and c;, with 
co # C1, and radii ro and 1, respectively. The background medium is assumed to 
have a speed of sound, co. 

The acoustic heterogeneity will cause the data function (ro, f) to differ from the 
ideal one g(ro, 7). The magnitude of this difference will be smaller, in general, for 


small values of ¢ than for large values of ¢. This can be understood by noting that, 


|ro—rl 
co 


in general, ty (r, ro) — will become larger as the path length through the 
speed-of-sound heterogeneity increases. This causes the surfaces of integration that 


contribute to ¢ (rg, f) to become less spherical for larger values of ¢. Accordingly, the 


€, Ultrasound 
77 receiver 


Fig. 5 <A speed-of-sound distribution comprised of two uniform concentric regions. Superim- 
posed on the figure are examples of how the surfaces of integration that contribution to the data 
function g(ro, f) are perturbed 
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Fig. 6 Images of a phantom object reconstructed from experimentally measured (a) full-time, (b) 
first half-time data functions (Reproduced from Anastasio et al. [4]) 


data function ¢(ro, 7) becomes less consistent with the spherical Radon transform 
model. 

The discussion above suggests that a half-time reconstruction method that 
employs ¢") (ro, 7) can produce images with reduced artifact and distortion levels 
than contained in images reconstructed from the complete, or full-time, data 
&(ro, f). An example of this is shown in Fig. 6. The data corresponded to a physical 
phantom study using a microwave source, as described in Ref. [61]. Measurements 
were taken at 160 equally spaced positions on the 2D circular scanning aperture of 
radius 70 mm, and for each measurement, the received pressure signal was sampled 
at 2,000 points, at a sampling frequency of 50 MHz. Images were reconstructed 
from full- and half-time data, via the EM algorithm as described in Ref. [4]. The 
contrast and resolution of the images reconstructed from half-time data appears to 
be superior to that of the images reconstructed from the full-time data. 


6 Discrete Imaging Models 


The imaging models discussed so far were expressed in their continuous forms. 
In practice, PAT imaging systems record temporal and spatial samples of p(ro, t), 
while the absorbed energy density is described by the function A(r). Accordingly, 
a realistic imaging model should be described mathematically as a continuous-to- 
discrete (C-D) mapping [8]. Moreover, when iterative reconstruction algorithms 
are employed, a discrete representation of A(r) is required to establish a suitable 
discrete-to-discrete approximate imaging model. In this section, we review these 
concepts within the context of PAT. 

The remainder of this section is organized as follows. In section “Continuous- 
to-Discrete Imaging Models,” we review the C—D versions of the continuous-to- 
continuous (C—C) models in Eqs. (21) and (22). Finite-dimensional object represen- 
tations are surveyed in section “Finite-Dimensional Object Representations” that 
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are used to establish the discrete-to-discrete (D—D) models in section “Discrete-to- 
Discrete Imaging Models.” In section “Iterative Image Reconstruction,” we briefly 
review some approaches to iterative image reconstruction that have been applied in 
PAT. The section concludes with a numerical example that demonstrates the effects 
of object representation error on image reconstruction accuracy. 


Continuous-to-Discrete Imaging Models 


In practice, p(ro,f) and g(ro,t) are discretized temporally and determined at 
a finite number of receiver locations. The vectors p,g € R will represent 
lexicographically ordered representations of the sampled data functions, where the 
dimension WN is defined by the product of the number of temporal samples acquired 
at each transducer location (S) and the number of transducer locations (M). Let 
Eqs. (21) and (22) be expressed in operator notation as 


P(ro,t) = HpA(r), (62) 
and 
g(¥o,t) = Hz A(r). (63) 


In general, a C—D operator can be interpreted as a discretization operator Dor 
acting on C—C operator Hcc [8]. Let y denote p or g and let Hcc denote H, or 
H,. The notation yj,; will be used to denote the nth element of the vector y. The 
C-D versions of Eqs. (21) and (22) can be expressed as 


y = DotHcc A(t) = Hcp A(t), (64) 


where D,, is discretization operator that characterizes the temporal and spatial 
sampling characteristics of the ultrasonic transducer. 

For the case where y = p and p(to,t) = H,A(r), Dor will be denoted as DY) 
and is defined as 


Pinsis) = [DY Peto. Ninsey = fat 0) f dQ plto.t)om(t0). (65) 
—oo Qo 


where m = 1,2,...,M is the index that specifies the mth transducer location 
Yom on the measurement aperture Qo, 5 = 1,2,...,S is the index of the time 
sample, and o,,(ro) and t;(¢) are functions that describe the spatial and temporal 
sampling apertures, respectively. They are determined by the sampling properties of 
ultrasonic transducers. In the ideal case, where both apertures are described by Dirac 
delta functions, the sth temporal sample for the mth transducer location represents 
the pressure at time sAT and location ro,,, where AT is the temporal sampling 
interval, that is, 
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P (mS+s] = P(fo.m ; sA T) : (66) 


We can express explicitly the C-D imaging model involving the pressure data as 
Pinsts) = [de AW) ins 4st), 67) 
V 
where V denotes the support volume of A(r) and 
CO 
hnses(t) = fata tte) f d%0 h(t, 20: t0)0m(t0) (68) 
—oo Qo 
defines a point response function. The kernel /(r, ro; fo) is defined as 
CO 
h(r, £0; to) -| dt I(t)G(r, ro; t, to), (69) 
—oo 


where /J(t) is the temporal illumination function and G(r, ro; tf, fo) is the Green’s 
function 


B dd(t) 
4nC,|r—Yro| dt 


G(r, Yo; f, to) = (70) 


Ir—rol 
t= 


By use of the singular value decomposition of the C—D operator in Eq. (67), a 
pseudoinverse solution can be computed numerically to estimate A(r) [8]. 

In order to establish a C-D imaging model involving the integrated pressure data, 
to first order, we can approximate the integral operator in Eq. (23) as 


AnC,cosAT < 
8[mS+s] = a a Yop [mS+q]- C7) 


gal 


For the case where y = g and g(ro,t) = H, A(r), Dox will be denoted as D\®) and 
is defined as 


&[mS+s] = [D® g(r, Dinsaa 


= sAT 3 [ia Tq(t) [4% Om oe (ee?) ; (72) 


q=l* 


Note that, in practice, g is not measured and is computed from the measured p by 
use of Eq.(71). Therefore, it is not physically meaningful to interpret g as being 
directly sampled from the raw measurement data. 
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Finite-Dimensional Object Representations 


When iterative image reconstruction algorithms are employed, a finite-dimensional 
representation of A(r) [8] is required. In this section, we review some finite- 
dimensional representations that have been employed in PAT. In the subsequent 
section, computer-simulation studies are conducted to demonstrate the effects of 
error in the object representation. 

An N-dimensional representation of A(r) can be described as 


N 
Aa(t) = )° Onjon(). (73) 


n=1 


where the subscript a indicates that A,(r) is an approximation of A(r). The 
functions @, (r) are called expansion functions and the expansion coefficients O,| are 
elements of the N-dimensional vector @. The goal of iterative image reconstruction 
methods is to estimate 6, for a fixed choice of the expansion functions ¢,(r). 

The most commonly employed expansion functions are simple image voxels 


1, if |x —Xnl,|y — yal. 1z— | < €/2 


74 
0, otherwise 4) 


gn (X,Y, 2) = 


where rr, = (Xn, Vn, Zn) Specify the coordinates of the nth grid point of a uniform 
Cartesian lattice and € defines the spacing between lattice points. 
In PAT, spherical expansion functions of the form 


Lif Je —xn)2 +O yn)? + Zn)? < €/2 


75 
0, otherwise (>) 


bn(X, Y,2) = 


have also proven to be useful [18,27]. The merit of this kind of expansion function 
is that the acoustic wave generated by each voxel can be calculated analytically. 
This facilitates determination of the system matrix utilized by iterative image 
reconstruction methods, as discussed below. Numerous other effective choices for 
the expansion functions [38] exist, including wavelets or other sets of functions that 
can yield sparse object representations [52]. 

In addition to an infinite number of choices for the expansion functions, there are 
an infinite number of ways to define the expansion coefficients 6. Some common 
choices include 


Veube 


Vvoxel 


On) = / d*x ¢,(r)A(r), (76) 
V 


or 


On) = far 6(r — rp) A(r). (77) 
V 
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For a given N, different choices of ¢, and @ will yield object representations that 
possess different representation errors 


$A(r) = A(r) — A, (rr). (78) 


An example of the effects of such representation errors on iterative reconstruction 
methods is provided in section “Iterative Image Reconstruction.” 


Discrete-to-Discrete Imaging Models 


Discrete-to-discrete (D-D) imaging models are required for iterative image 
reconstruction. These can be obtained systematically by substitution of a finite- 
dimensional object representation into the C-D imaging model in Eq. (64): 


N 
Ya = HcpAa(r) = >) OjHev{gn(r)} = HO, (79) 


n=1 


where the D—-D operator H is commonly referred to as the system matrix. The 
system matrix H is of dimension (MS) x N, and an element of H will be denoted 
by Hjn.m. Note that the data vector y, # y, due to the fact that a finite-dimensional 
approximate object representation was employed. In other words, y, represents an 
approximation of the measured pressure data, denoted by p,, or the corresponding 
approximate integrated pressure data g,. 

For the case where yz = py, the system matrix H will be denoted as H™) and its 
elements are defined as 


HO), = / dr Gn ()hins 45(8) = D”{ pa (to, t0)}, (80) 
V 


where h»,5+5(r) is defined in Eq. (68) and 


Pa(tosto) = fdr dy (eyh(r 400). (81) 


Equation (80) provides a clear two-step procedure for computing the system 
matrix. First, p, (10, fo) is computed. Physically, this represents the pressure data, in 
its continuous form, received by an ideal point transducer when the absorbing object 
corresponds to ¢,,(r). Secondly, a discretization operator is applied that samples the 
ideal data and degrades it by the transducer response. Alternatively, the elements of 
the system matrix can be measured experimentally by scanning an object whose 
form matches the expansion functions through the object volume and recording 
the resulting pressure signal at each transducer location ro,,, for each value of n 
(location of expansion function), at time intervals sA7. For the case of spherical 
expansion elements, this approach was implemented in [18]. 
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This two-step approach for determining H be formulated as 


H = Sok, (82) 


6699 


where “o” denotes an element-wise product. Each element of Ho is defined as 
Aon s +s,n] = Pn (Yom; SAT). (83) 


The MS x N matrix S can be interpreted as a sensitivity map, whose elements are 
defined as 


Dot {Pn (To, to) 


“Pn(tom,SAT) ” 84 
Pn (Fon , sAT) (84) 


StmS+s.n] = 


For the case where y, = g,, similar interpretations hold. The system matrix H 
will be denoted as H“), and its elements are defined as 


Hea = DE 2, (To, to), (85) 
where 
4nCycoto f® 
8n (To, to) = ee: : | ato | d°x bn (r)h(r, ¥0; &). (86) 
0 V 


Numerical Example: Impact of Representation Error on Computed 
Pressure Data 

Consider a uniform sphere of radius R; = 5mm as the optical absorber (acoustic 
source). Assuming Dirac delta (i.e., ideal) temporal and spatial sampling, the 
pressure data were computed at a measurement location rp 65 mm away from the 
center of the sphere by use of D—D and C-C imaging models. For the uniform 
sphere, the pressure waveform can be computed analytically as 

d B 


sAT) = — | ——— 
Pity, SAT) dt | 4aC)ycot 


g(Fo. | 


t=sAT 


II 


| see Ito — rel — cosAT), if |CosAT = [to — rl] < Ry 
0, otherwise 

(87) 
where r, is the center of the spherical source and AT is the sampling interval. 
As discussed in section “The Thermoacoustic Effect and Signal Generation,” the 
pressure possesses an “N” shape waveform as shown as the dashed red curve 
in Fig. 7. Finite-dimensional object representations of the object were obtained 
according to Eq. (73) with ¢,(r) corresponding to the uniform spheres described 
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— 256° 
— 643 


39.35 41.35 43.35 45.35 
Time (us) 


Fig. 7 Pressure data generated by continuous imaging model (red dash) and discrete imaging 
model using 256 X 256 X 256 voxels (blue solid) and 64 x 64 x 64 voxels (green solid) 


in Eq.(75). The expansion coefficients were computed according to Eq. (76). 
Two approximate object representations were considered. The first representation 
employed N = 256° spherical expansion functions of radius 0.04 mm, while the 
second employed N = 64? expansion functions of radius 0.16 mm. The resulting 
pressure signals are shown as Fig.7, where the speed of sound cp = 1.521 mm/s 
and AT = 0.05 ws. As expected, the error in the computed pressure data increases 
as the voxel size is increased. In practice, this error would represent a data 
inconsistency between the measured data and the assumed D—D imaging model, 
which can result in image artifacts as demonstrated by the example below. 


Iterative Image Reconstruction 


Once the system matrix H is determined, as described in the previous section, an 
estimate of A(r) can be computed in two distinct steps. First, from knowledge of 
the measured data and system matrix, Eq. (79) is inverted to estimate the expansion 
coefficients @. Second, the estimated expansion coefficients are employed with 
Eq. (73) to determine the finite-dimensional approximation A,(r). Each of steps 
introduces error into the final estimate of A(r). In the first step, due to noise in 
the measured data y,, modeling errors in H, and/or if H is not full rank, the true 
values coefficients 9 cannot generally be determined. The estimated 6 will therefore 
depend on the definition of the approximate solution and the particular numerical 
algorithm used to determine it. Even if 8 could somehow be determined exactly, 
the second step would introduce error due to the approximate finite-dimensional 
representation of A(r) employed. This error is influenced by the choice of N and 
on (r) and is object dependent. 
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> 0.75 


0.5 


0.25 


Fig. 8 The 2D numerical phantom @ representing the object function A(r) 


Due to the large size of H, iterative methods are often employed to estimate 
@. Iterative approaches offer a fundamental and flexible way to incorporate a prior 
information regarding the object, to improve the accuracy of the estimated @. A vast 
literature on iterative image reconstruction methods exists [7, 20,21, 60], which we 
leave to the reader to explore. Examples of applications of iterative reconstruction 
methods in PAT are described in references [1,4,5, 18,48, 69]. A numerical example 
demonstrating how object representation error can affect the accuracy of iterative 
image reconstruction is provided next. 


Numerical Example: Influence of Representation Error on Image 
Accuracy 

We assume focused transducers are employed that receive only acoustic pressure 
signals transmitted from the imaging plane and therefore the 3D spherical Radon 
transform imaging model to a 2D circular mean model. A 2D phantom comprised 
of uniform disks possessing different gray levels, radii, and locations was assumed 
to represent A(r). The radius of the phantom was 1.0 (arbitrary units). A finite- 
dimensional representation A, (r) was formed according to Eq. (73), with N = 2567 
and @¢,,(r) chosen to be conventional pixels described by a 2D version of Eq. (74). 
The expansion coefficients ,; were computed by use of Eq. (77). Figure 8 displays 
the computed expansion coefficient vector 0 that has been reshaped into a 256 x 256 
for display purposes. 

A circular measurement aperture Qo of radius 1.2 that enclosed the object was 
employed. At each of 360 uniformly, spaced transducer locations, ro), on the 
measurement circle, simulated pressure data p, were computed from the integrated 
data g by use of the formula 
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80 160 240 
Pixel 


Fig. 9 Images reconstructed by the least-squares conjugate gradient algorithm from pressure data 
obtained by (a) numerical imaging model and (b) analytical imaging model. (c) Vertical profiles 
through the center of subfigure (a) (solid blue), subfigure (b) (solid green), and Fig. 8 (dashed red) 


(88) 


= B | feseee/t + 1) _ Zims+s—1/(S _ ) 
PalmS+s] = : 


An Cpco 2AT2 


Two versions of the pressure data were computed, corresponding to the cases where 
g was computed analytically or by use of the assumed D—D imaging model. These 
simulated pressure data are denoted by pa””” and p”“", respectively. At each 
transducer location, 300 temporal samples of p(ro, f) were computed. Accordingly, 
the pressure vector py was a column vector of length 360 x 300. 

The conjugate gradient algorithm was employed to find the least-squares esti- 


mate 6, 


6 = argmin||p, — HO ||", (89) 
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ly . . 
where py = Pa” or p””". For the noiseless data, the images reconstructed 


from pa” and p”“" after 150 iterations are shown as Fig. 9a, b, respectively. 


The image reconstructed from the data p/”” is free of significant artifacts and 
is nearly identical to the original object. This is expected because the finite- 
dimensional object representation was used to produce the simulated measurement 
data and establish the system matrix, and therefore the system of equations in 
Eq. (79) is consistent. Generating simulation data in this way would constitute an 
“inverse crime.” Conversely, the image reconstructed from the data pa"“”’ contained 
high-frequency artifacts due to the fact that the system of equations in Eq. (79) 
is inconsistent. The error in the reconstructed images could be minimized by 
increasing the dimension of the approximate object representation. This simple 
example demonstrates the importance of carefully choosing a finite-dimensional 
object representation in iterative image reconstruction. 


7 Conclusion 


Photoacoustic tomography is a rapidly emerging biomedical imaging modality 
that possesses many challenges for image reconstruction. In this chapter, we have 
reviewed the physical principles of PAT. Contrast mechanisms in PAT were dis- 
cussed, and the imaging models that relate the measured photoacoustic wavefields 
to the sought-after optical absorption distribution were described in their continuous 
and discrete forms. 
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Abstract 

The chapter surveys the mathematical models, problems, and algorithms of the 
thermoacoustic tomography (TAT) and photoacoustic tomography (PAT). TAT 
and PAT represent probably the most developed of the several novel “hybrid” 
methods of medical imaging. These new modalities combine different physical 
types of waves (electromagnetic and acoustic in case of TAT and PAT) in such 
a way that the resolution and contrast of the resulting method are much higher 
than those achievable using only acoustic or electromagnetic measurements. 


1 Introduction 


We provide here just a very brief description of the thermoacoustic tomogra- 
phy/photoacoustic tomography (TAT/PAT) procedure, since the relevant physics and 
biology details can be found in another chapter [94] in this volume, as well as in 
the surveys and books [93,95]. In TAT (PAT), a short pulse of radio-frequency 
EM wave (correspondingly, laser beam) irradiates a biological object (e.g., in the 
most common application, human breast), thus causing small levels of heating. The 
resulting thermoelastic expansion generates a pressure wave that starts propagating 
through the object. The absorbed EM energy and the initial pressure it creates are 
much higher in the cancerous cells than in healthy tissues (see the discussion of 
this effect in [93-95]). Thus, if one could reconstruct the initial pressure f(x), 
the resulting TAT tomogram would contain highly useful diagnostic information. 
The data for such a reconstruction are obtained by measuring time-dependent 
pressure p(x,f) using acoustic transducers located on a surface S (we will call 
it the observation or acquisition surface) completely or partially surrounding the 
body (see Fig. 1). Thus, although the initial irradiation is electromagnetic, the actual 
reconstruction is based on acoustic measurements. As a result, the high contrast 
is produced due to a much higher absorption of EM energy by cancerous cells 
(ultrasound alone would not produce good contrast in this case), while the good 
(submillimeter) resolution is achieved by using ultrasound measurements (the radio- 
frequency EM waves are too long for high-resolution imaging). Thus, TAT, by using 
two types of waves, combines their advantages, while eliminating their individual 
deficiencies. 

The physical principle upon which TAT/PAT is based was discovered by Alexan- 
der Graham Bell in 1880 [19], and its application for imaging of biological tissues 
was suggested a century later [21]. It began to be developed as a viable medical 
imaging technique in the middle of the 1990s [53, 69]. 

Some of the mathematical foundations of this imaging modality were originally 
developed starting in the 1990s for the purposes of the approximation theory, 
integral geometry, and sonar and radar (see [4,7,38,54,60] for references and exten- 
sive reviews of the resulting developments). Physical, biological, and mathematical 
aspects of TAT/PAT have been recently reviewed in [4, 38, 39, 54, 70, 89, 92,93, 95]. 

TAT/PAT is just one, probably the most advanced at the moment, example of 
the several recently introduced hybrid imaging methods, which combine different 
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types of radiation to yield high quality of imaging unobtainable by single-radiation 
modalities (see [10, 11,40, 55,95] for other examples). 


2 Mathematical Models of TAT 


In this section, we describe the commonly accepted mathematical model of the TAT 
procedure and the main mathematical problems that need to be addressed. Since 
for all our purposes PAT results in the same mathematical model (although the 
biological features that TAT and PAT detect are different; see details in Ref. [13]), 
we will refer to TAT only. 


Point Detectors and the Wave Equation Model 


We will mainly assume that point-like omnidirectional ultrasound transducers, 
located throughout an observation (acquisition) surface S, are used to detect the 
values of the pressure p(y,f), where y € S is a detector location and t > 0 
is the time of the observation. We also denote by c(x) the speed of sound at a 
location x. Then, it has been argued that the following model describes correctly 
the propagating pressure wave p(x,t) generated during the TAT procedure (e.g., 
[13, 31, 88, 94, 97]): 


Pu=c?(x)Axp, t>0,x eR? 


p(x,0) = f(x), pr(x, 0) = 0. (1) 


Here f(x) is the initial value of the acoustic pressure, which one needs to find in 
order to create the TAT image. In the case of a closed acquisition surface S', we will 
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Fig. 2. The observation surface S and the domain Q containing the object to be imaged 


denote by Q the interior domain it bounds. Notice that in TAT the function f(x) is 
naturally supported inside (2. We will see that this assumption about the support of 
jf sometimes becomes crucial for the feasibility of reconstruction, although some 
issues can be resolved even if f has nonzero parts outside the acquisition surface. 

The data obtained by the point detectors located on a surface S are represented 
by the function 


g(y,t)i= p(y,t) forye S,t> 0. (2) 
Figure 2 illustrates the space-time geometry of (1). 


We will incorporate the measured data g into the system (1), rewriting it as 
follows: 


Pu =C(x)Ayp, t>0,x ER? 
P(x, 0) = f(x), pr(x,0) =0 (3) 
pis =ely.t), (the S xR. 
Thus, the goal in TAT/PAT is to find, using the data g(y,f) measured by 
transducers, the initial value f(x) at t = 0 of the solution p(x, t) of (3). 


We will use the following notation: 


Definition 1. We will denote by W the forward operator 


W: f(x) gy, 0), (4) 


where f and g are described in (3). 
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Remark 1. 

e The reader should notice that if a different type of detector is used, the system (1) 
will still hold, while the measured data will be represented differently from (2) 
(see section “Variations on the Theme: Planar, Linear, and Circular Integrating 
Detectors’). This will correspondingly influence the reconstruction procedures. 

¢ We can consider the same problem in the space R” of any dimension, not just 
in 3D. This is not merely a mathematical abstraction. Indeed, in the case of the 
so-called integrating line detectors (section “Variations on the Theme: Planar, 
Linear, and Circular Integrating Detectors”), one deals with the 2D situation. 


Acoustically Homogeneous Media and Spherical Means 


If the medium being imaged is acoustically homogeneous (i.e., c(x) equals to a 
constant, which we will assume to be equal to | in appropriate units), as it is 
approximately the case in breast imaging, one deals with the constant coefficient 
wave equation problem 


Pu = AxP, t>0,xeR 
P(x, 0) = f(x), pr(x,0) = 0 (5) 
pls =8y.t), (y.t)eSxR?. 


In this case, the well-known Poisson—Kirchhoff formulas [27, Chap. VI, Sect. 13.2, 
Formula (15)] for the solution of the wave equation give in 3D 


) 
PO, t) =a (R/O), (6) 


where 


RNGM=— f fot rdyy @) 


lyl=1 


is the spherical mean operator applied to the function f(x), dA is the standard area 
element on the unit sphere in R3, and a is a constant. (Versions in all dimensions are 
known; see (16) and (15).) One can derive from here that knowledge of the function 
g(x,t) for x € S andallt > Ois equivalent to knowing the spherical mean Rf (x, t) 
of the function f for any points x € S and any ¢t > 0. One thus needs to study the 
spherical mean operator R : f — Rf, or, more precisely, its restriction to the 
points x € S only, which we will denote by M: 


M f(x, t) := = i f(x+ty)dA(y), x€S,t>0. (8) 


lyl=1 
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Due to the connection between the spherical mean operator and the wave equation, 
one can choose to work with the former, and in fact many works on TAT do 
so. The spherical mean operator M resembles the classical Radon transform, the 
common tool of computed tomography [63], which integrates functions over planes 
rather than spheres. This analogy with Radon transform, although often purely 
ideological, rather than technical, provides important intuition and frequently points 
in reasonable directions of study. However, when the medium cannot be assumed to 
be acoustically homogeneous, and thus c(x) is not constant, the relation between 
TAT and integral geometric transforms, such as the Radon and spherical mean 
transforms to a large extent breaks down, and thus one has to work with the wave 
equation directly. 

In what follows, we will address both models of TAT (the PDE model and the 
integral geometry model) and thus will deal with both forward operators W and M. 


Main Mathematical Problems Arising in TAT 


We now formulate a list of problems related to TAT, which will be addressed in 
detail in the rest of the article. (This list is more or less standard for a tomographic 
imaging method.) 


Sufficiency of the data: The first natural question to ask is as follows: Is the data 

collected on the observation surface S' sufficient for the unique reconstruction 
of the initial pressure f(x) in (3)? In other words, is the kernel of the forward 
operator W zero? Or, to put it differently, for which sets S € R°* the data 
collected by transducers placed along S determines f uniquely? Yet another 
interpretation of this question is through observability of solutions of the wave 
equation on the set S: does observation on S' of a solution of the problem (1) 
determine the solution uniquely? 
When the speed of sound is constant, and thus the spherical mean model 
applies, the equivalent question is whether the operator M has zero kernel on an 
appropriate class of functions (say, continuous functions with compact support). 
As it is explained in [7], the choice of precise conditions on the local function 
class, such as continuity, is of no importance for the answer to the uniqueness 
question, while behavior at infinity (e.g., compactness of support) is. So, 
without loss of generality, when discussing uniqueness, one can assume f(x) 
in (3) to be infinitely differentiable. 

Inversion formulas and algorithms: Since a practitioner needs to see the actual 
tomogram, rather than just know its existence, the next natural question arises: If 
uniqueness the data collected on S' is established, what are the actual inversion 
formulas or algorithms? Here again one can work with smooth functions, in the 
end extending the formulas by continuity to a wider class. 

Stability of reconstruction: If we can invert the transform and reconstruct f from 
the data g, how stable is the inversion? The measured data are unavoidably 
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corrupted by errors, and stability means that small errors in the data lead to 
only small errors in the reconstructed tomogram. 

Incomplete data problems: What happens if the data is “incomplete,” for instance, 
if one can only partially surround the object by transducers? Does this lead 
to any specific deterioration in the tomogram and if yes, to what kind of 
deterioration? 

Range descriptions: The next question is known to be important for the analysis of 
tomographic problems: What is the range of the forward operator W : f te g 
that maps the unknown function f to the measured data g? In other words, what 
is the space of all possible “ideal” data g(t, y) collected on the surface S'’? In the 
constant speed of sound case, this is equivalent to the question of describing the 
range of the spherical mean operator M in appropriate function spaces. Such 
ranges often have infinite co-dimensions, and the importance of knowing the 
range of Radon type transforms for analyzing problems of tomography is well 
known. For instance, such information is used to improve inversion algorithms, 
complete incomplete data, and discover and compensate for certain data errors 
(e.g., [41, 45, 63, 68, 70] and references therein). In TAT, range descriptions are 
also closely connected with the speed of sound determination problem listed 
next (see section “Reconstruction of the Speed of Sound” for a discussion of 
this connection). 

Speed of sound reconstruction: As the reader can expect, reconstruction proce- 
dures require the knowledge of the speed of sound c(x). Thus, the problem 
arises of the recovery of c(x) either from an additional scan or (preferably) 
from the same TAT data. 


Variations on the Theme: Planar, Linear, and Circular Integrating 
Detectors 


In the most basic and well-studied version of TAT described above, one utilizes 
point-like broadband transducers to measure the acoustic wave on a surface sur- 
rounding the object of interest. The corresponding mathematical model is described 
by the system (3). In practice, the transducers cannot be made small enough, 
since smaller detectors yield weaker signals resulting in low signal-to-noise ratios. 
Smaller transducers are also more difficult to manufacture. 

Since finite size of the transducers limits the resolution of the reconstructed 
images, researchers have been trying to design alternative acquisition schemes using 
receivers that are very thin but long or wide. Such are 2D planar detectors [24, 43] 
and 1D linear and circular [23, 42, 73, 103] detectors. 

We will assume throughout this section that the speed of sound c(x) is constant 
and equal to 1. 

Planar detectors are made from a thin piezoelectric polymer film glued onto a 
flat substrate (see, e.g., [75]). Let us assume that the object is contained within 
the sphere of radius R. If the diameter of the planar detector is sufficiently large 
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(see [75] for details), it can be assumed to be infinite. The mathematical model 
of such an acquisition technique is no longer described by (3). Let us define 
the detector plane II(s,w) by equation x-w = s, where o is the unit normal 
to the plane and s is the (signed) distance from the origin to the plane. Then, 
while the propagation of acoustic waves is still modeled by (1), the measured data 
Zplanar(S,¢,@) (up to a constant factor which we will, for simplicity, assume to be 
equal to 1) can be represented by the following integral: 


Splanar(S, @, t) _ / D(x, t)dA(x) 
TI (s,@) 


where d A(x) is the surface measure on the plane. Obviously, 
Splanar(S,@,0) = / D(x, 0)dA(x) = / f(x)dA(x) = F(s, a), 
T(s,w) TI(s,@) 
i.e., the value of g at t = 0 coincides with the integral F'(s, w) of the initial pressure 
F(x) over the plane II(s, w) orthogonal to w. 


One can show [24, 43] that for a fixed w, function gplanar(s, @, ¢) is the solution 
to 1D wave equation 


and thus 
Splanar(S, Oo, t) = [ Splanas (i; o,s — t) + Splanar(S, @,S + t)| 


[F(s+t,w)+ F(s—t,o)]. 


Nl NI eRe 


Since the detector can only be placed outside the object, i.e., s > R, the term F(s + 
t,@) vanishes, and one obtains 


Splanar(S, @, ¢) = F(s —t,@). 


In other words, by measuring 2planar(S, @, 1), one can obtain values of the planar 
integrals of f(x). If, as proposed in [24, 43], one conducts measurements for all 
planes tangent to the upper half sphere of radius R (i.e.,5 = R,w € $2), then the 
resulting data yield all values of the standard Radon transform of f(x). Now the 
reconstruction can be carried out using one of the many known inversion algorithms 
for the latter transform [63]. 

Linear detectors are based on optical detection of acoustic signal. Some of the 
proposed optical detection schemes utilize as the sensitive element a thin straight 
optical fiber in combination with Fabry—Perot interferometer [23, 42]. Changes of 
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acoustic pressure on the fiber change (proportionally) its length; this elongation, 
in turn, is detected by the interferometer. A similar idea is used in [73]; in this 
work the role of a sensitive element is played by a laser beam passing through the 
water in which the object of interest is submerged, and thus the measurement does 
not perturb the acoustic wave. In both cases, the length of the sensitive element 
exceeds the size of the object, while the diameter of the fiber (or of the laser beam) 
can be made extremely small (see [75] for a detailed discussion), which removes 
restrictions on resolution one can achieve in the images. 

Let us assume that the fiber (or laser beam) is aligned along the line 
I (51,82, @1,@2) = {x|x = 5,@ +52@2 + sw}, where vectors w;, @2, and w form an 
orthonormal basis in R*. Then the measured quantities 2jinear(S1,52,@1,@2,t) are 
equal (up to a constant factor which, we will assume, equals to 1) to the following 
line integral: 


linear ($1, 52, @1, @2,¢) = / DP(s1@, + S2@2 + sw,t)ds. 


R! 


Similar to the case of planar detection, one can show [23,42,73] that for fixed vectors 
@ , @2 the measurements Qijnear(S1, 52, @1, @2, t) satisfy the 2D wave equation 


Pg dag wg 
ds? ss Ot?" 


The initial values gjinear(S1, 52, @1,@2,0) coincide with the line integrals of f(x) 
along lines /(s1,52,@ ,@2). Suppose one makes measurements for all values of 
51(T), S2(t) corresponding to a curve y = {x|x = s,(t)@; +.52(t)an2, |] < Tt < T} 
lying in the plane spanned by @,, w2. Then one can try to reconstruct the initial value 
of g from the values of g on y. This problem is a 2D version of (3), and thus the 
known algorithms (see Sect. 4) are applicable. 

In order to complete the reconstruction from data obtained using line detectors, 
the measurements should be repeated with different directions of w. For each value 
of w, the 2D problem is solved; the solutions of these problems yield values of line 
integrals of f(x). If this is done for all values of w lying on a half circle, the set 
of the recovered line integrals of f(x) is sufficient for reconstructing this function. 
Such a reconstruction represents the inversion of the well known in tomography X- 
ray transform. The corresponding theory and algorithms can be found, for instance, 
in [63]. 

Finally, the use of circular integrating detectors was considered in [103]. Such a 
detector can be made out of optical fiber combined with an interferometer. In [103], 
a closed-form solution of the corresponding inverse problem is found. However, this 
approach is very new, and neither numerical examples nor reconstructions from real 
data have been obtained yet. 
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3 Mathematical Analysis of the Problem 


In this section, we will address most of the issues described in section “Main 
Mathematical Problems Arising in TAT,’ except the reconstruction algorithms, 
which will be discussed in Sect. 4. 


Uniqueness of Reconstruction 


The problem discussed here is the most basic one for tomography: Given an 
acquisition surface S along which we distribute detectors, is the data g(y,t) for 
y € S,t > 0 (see (3)) sufficient for a unique reconstruction of the tomogram f? A 
simple counting of variables shows that S' should be a hypersurface in the ambient 
space (i.e., a surface in R? or a curve in R). As we will see below, although there 
are some simple counterexamples and remaining open problems, for all practical 
purposes, the uniqueness problem is positively resolved, and most surfaces S do 
provide uniqueness. We address this issue for acoustically homogeneous media first 
and then switch to the variable speed case. 

Before doing so, however, we would like to dispel a concern that arises when 
one looks at the problem of recovering f from g in (3). Namely, an impression 
might arise that we consider an initial-boundary value (IBV) problem for the wave 
equation in the cylinder Q x Rt, and the goal is to recover the initial data f 
from the known boundary data g. This is clearly impossible, since according to 
standard PDE theorems (e.g., [27]), one can solve this IBV problem for arbitrary 
choice of the initial data f and boundary data g (as long as they satisfy simple 
compatibility conditions, which are fulfilled, for instance, if f vanishes near S 
and g vanishes for small t, which is the case in TAT). This means that apparently 
g contains essentially no information about f at all. This argument, however, is 
flawed, since the wave equation in (3) holds in the whole space, not just in Q. In 
other words, S' is not a boundary, but rather an observation surface. In particular, 
considering the wave equation in the exterior of S, one can derive that if f is 
supported inside Q, the boundary values g of the solution p of (3) also determine 
the normal derivative of p at S for all positive times. Thus, we in fact have (at 
least theoretically) the full Cauchy data of the solution p on S, which should be 
sufficient for reconstruction. Another way of addressing this issue is to notice that 
if the speed of sound is constant, or at least non-trapping (see the definition below 
in section “Acoustically Inhomogeneous Media’), the energy of the solution in any 
bounded domain (in particular, in (2) must decay in time. The decay when t > oo 
together with the boundary data g guarantees the uniqueness of solution and thus 
uniqueness of recovery /f. 

These arguments, as the reader will see, play a role in understanding reconstruc- 
tion procedures. 
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Acoustically Homogeneous Media 
We assume here the sound speed c(x) to be constant (in appropriate units, one can 
choose it to be equal to 1, which we will do to simplify considerations). 

In order to state the first important result on uniqueness, let us recall the system 
(5), allowing an arbitrary dimension n of the space: 


Pu =Axp, t>0,x eR’ 
P(x, 0) = f(x), pr(x,0) =0 (9) 
pls =8y.t), (y.t)eSxR?. 


We introduce the following useful definition: 


Definition 2. A set S is said to be a uniqueness set if when used as the acquisition 
surface, it provides sufficient data for unique reconstruction of the compactly 
supported tomogram ff (i.e., the observed data g in (9) determines uniquely function 
J). Otherwise, it is called a nonuniqueness set. 


In other words, S is a uniqueness set if the forward operator W (or, equivalently, 
My) has zero kernel. 

We have not indicated above the smoothness class of f(x). However, it is not 
hard to show (e.g., [7]) that the uniqueness does not depend on the smoothness of 
};; for simplicity, the reader can assume that /f is infinitely differentiable. On the 
other hand, compactness of support is important in what follows. 

We will start with a very general statement about the acquisition (observation) 
sets § that provide insufficient information for unique reconstruction of f (see [7] 
for the proof and references): 


Theorem 1. /f S is a nonuniqueness set, then there exists a nonzero harmonic 
polynomial Q, which vanishes on S. 


This theorem implies, in particular, that all “bad” (nonuniqueness) observation 
sets are algebraic, i.e., they have a polynomial vanishing on them. Turning this 
statement around, we conclude that any set S that is a uniqueness set for harmonic 
polynomials is sufficient for unique TAT reconstruction (although, as we will see in 
section “Incomplete Data,” this does not mean practicality of the reconstruction). 

The proof of Theorem 1, which the reader can find in [7,54], is not hard and 
in fact is enlightening, but providing it would lead us too far from the topic of this 
survey. 

We will consider first the case of closed acquisition surfaces, i.e., the ones that 
completely surround the object to be imaged. We will address the general situation 
afterward. 
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Closed Acquisition Surfaces S 

Theorem 2 ([7]). If the acquisition surface S is the boundary of bounded domain 
Q (i.e., a closed surface), then it is a uniqueness set. Thus, the observed data g in 
(9) determines uniquely the sought function f € Te ae (R"). (The statement holds, 


even though f is not required to be supported inside S.) 


Proof. Indeed, since there are no nonzero harmonic functions vanishing on a closed 
surface S, Theorem | implies Theorem 2. i 


There is, however, another more intuitive explanation of why Theorem 2 holds 
true (although it requires somewhat stronger assumptions or a more delicate proof 
than the one indicated below). Namely, since the solution p of (9) has compactly 
supported initial data, its energy is decaying inside any bounded domain, in 
particular inside Q (see section “Acoustically Inhomogeneous Media” and [32,47] 
and references therein about local energy decay). On the other hand, if there is 
nonuniqueness, there exists a nonzero f such that g(y,t) = 0 forall y € S andt. 
This means that we can add homogeneous Dirichlet boundary conditions p |s= 0 
to (9). But then the standard PDE theorems [27] imply that the energy stays constant 
in 2. Combination of the two conclusions means that p is zero in Q for all times f. 
It is well known [27] that such a solution of the wave equation must be identically 
zero everywhere, and thus f = 0. 

This energy decay consideration can be extended to some classes of non- 
compactly supported functions f of the L? classes, leading to the following result 
of [1]: 


Theorem 3 ([1]). Let S be the boundary of a bounded domain in R" and f € 
L?(R"). Then: 


l. ifp< an and the spherical mean of f over almost every sphere centered on S 
is equal to zero, then f = 0. 
2. The previous statement fails when p > a and § is a sphere. 


In other words, a closed surface S is a uniqueness set for functions f € L?(R") 
2n 


when p < a and might fail to be such when p > 5. 
This result shows that the assumption, if not necessarily of compactness of 
support of f, but at least of a sufficiently fast decay of f at infinity, is important for 


the uniqueness to hold. 
General Acquisition Sets S 
Theorems | and 2 imply the following useful statement: 


Theorem 4. [fa set S is not algebraic, or if it contains an open part of a closed 
analytic surface 1, then it is a uniqueness set. 
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2=N 


Fig. 3 Coxeter cross of N lines 


Indeed, the first claim follows immediately from Theorem |. The second one 
works out as follows: if an open subset of an analytic surface I’ is a nonuniqueness 
set, then by an analytic continuation type argument (see [7]), one can show that the 
whole I is such a set. However, this is impossible, due to Theorem 2. 

There are simple examples of nonuniqueness surfaces. Indeed, if S is a plane 
in 3D (or a line in 2D or a hyperplane in dimension n) and f(x) in (3) is odd 
with respect to S, then clearly the whole solution of (3) has the same parity and 
thus vanishes on S for all times ¢. This means that, if one places transducers on a 
planar S, they might register zero signals at all times, while the function f to be 
reconstructed is not zero. Thus, there is no uniqueness of reconstruction when S' is 
a plane. On the other hand (see [27,51]), if f is supported completely on one side 
of the plane S (the standard situation in TAT), it is uniquely recoverable from its 
spherical means centered on S and thus from the observed data g. 

The question arises on what are other “bad” (nonuniqueness) acquisition surfaces 
than planes. This issue has been resolved in 2D only. Namely, consider a set of N 
lines on the plane intersecting at a point and forming at this point equal angles. We 
will call such a figure the Coxeter cross Ly (see Fig.3). It is easy to construct a 
compactly supported function that is odd simultaneously with respect of all lines 
in Uy. Thus, a Coxeter cross is also a nonuniqueness set. The following result, 
conjectured in [60] and proven in the full generality in [7], shows that up to adding 
finitely many points, this is all that can happen to nonuniqueness sets: 


Theorem 5 ([7]). A set S in the plane R? is a nonuniqueness set for compactly 
supported functions f, if and only if it belongs to the union Xy \) ® of a Coxeter 
cross Uy and a finite set of points ®. 
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Fig. 4 The conjectured structure of a most general nonuniqueness set in 3D 


Again, compactness of support is crucial for the proof provided in [7]. There are 
no other proofs known at the moment of this result (see the corresponding open 
problem in Sect.5). In particular, there is no proven analogue of Theorem 3 for 
non-closed sets S (unless S is an open part of a closed analytic surface). 

The n-dimensional (in particular, 3D) analogue of Theorem 5 has been conjec- 
tured [7], but never proven, although some partial advances in this direction have 
been made in [8, 36]. 


Conjecture 1. A set S in R” is a nonuniqueness set for compactly supported 
functions f, if and only if it belongs to the union ¥ |) ®, where » is the cone of 
zeros of a homogeneous (with respect to some point in R”) harmonic polynomial, 
and ® is an algebraic subset of IR” of dimension at most n — 2 (see Fig. 4). 


Uniqueness Results for a Finite Observation Time 

So far, we have addressed only the question of uniqueness of reconstruction in 
the nonpractical case of the infinite observation time. There are, however, results 
that guarantee the uniqueness of reconstruction for a finite time of observation. The 
general idea is that it is sufficient to observe for the time that it takes the geometric 
rays (see section “Acoustically Inhomogeneous Media”) from the interior Q of S 
to reach S. In the case of a constant speed, which we will assume to be equal to 1, 
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the rays are straight and are traversed with the unit speed. This means that if D is 
the diameter of Q (i.e., the maximal distance between two points in the closure of 
(2), then after time t = D, all rays coming from Q have left the domain. Thus, one 
hopes that waiting till time t = D might be sufficient. In fact, due to the specific 
initial conditions in (3), namely, that the time derivative of the pressure is equal to 
zero at the initial moment, each singularity of # emanates two rays, and at least one 
of them will reach S in time not exceeding D/2. And indeed, the following result 
of [36] holds: 


Theorem 6 ([36]). Jf S is smooth and closed surface bounding domain Q and D 
is the diameter of Q, then the TAT data on S collected for the time 0 < t < 0.5D 
uniquely determines f. 


Notice that a shorter collection time does not guarantee uniqueness. Indeed, if 
S is a sphere and the observation time is less than 0.5D, due to the finite speed of 
propagation, no information from a neighborhood of the center can reach S during 
observation. Thus, values of f in this neighborhood cannot be reconstructed. 


Acoustically Inhomogeneous Media 
We assume that the speed of sound is strictly positive, c(x) > c > 0, and such that 
c(x) — 1 has compact support, i.e., c(x) = 1 for large x. 


Trapping and Non-trapping 
We will frequently impose the so-called non-trapping condition on the speed of 
sound c(x) in R”. To introduce it, let us consider the Hamiltonian system in RY. 


with the Hamiltonian H = eOig |? 


xf = = ae 


gj = HE = -49 (0) IBF vs 


X|lr=0 = X0, €Elr=o0 = &o- 


The solutions of this system are called bicharacteristics, and their projections into 
IR% are rays (or geometric rays). 


Definition 3. We say that the speed of sound c(x) satisfies the non-trapping 
condition, if all rays with & 4 0 tend to infinity when t > oo. 
The rays that do not tend to infinity are called trapped. 


A simple example, where quite a few rays are trapped, is the radial parabolic 
sound speed c(x) = c|x|?. 

It is well known (e.g., [46]) that singularities of solutions of the wave equation 
are carried by geometric rays. In order to make this statement more precise, we need 
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to recall the notion of a wave front set WF (u) of a distribution u(x) in R”. This set 
carries detailed information on singularities of u(x). 


Definition 4. Distribution u(x) is said to be microlocally smooth near a point 
(Xo, &), where xo, & € IR”, and & + 0, if there is a smooth “cutoff” function (x) 
such that @(x9) # 0 and that the Fourier transform pu(é) of the function @(x)u(x) 
decays faster than any power |€|~" when |&| —> 00, in directions that are close 
to the direction of &. (We remind the reader that if this Fourier transform decays 
that way in all directions, then u(x) is smooth (infinitely differentiable) near the 
point Xo.) 

The wave front set WF(u) C Ri x (Ry \0) of u consists of all pairs (xo, 0) 


such that u is not microlocally smooth near (Xo, &). 


In other words, if (xo, &)) € WF(u), then u is not smooth near xo, and the 
direction of & indicates why it is not: the Fourier transform does not decay well 
in this direction. For instance, if u(x) consists of two smooth pieces joined non- 
smoothly across a smooth interface &, then WF(u) can only contain pairs (x, &) 
such that x € & and & is normal to © at x. 

It is known that the wave front sets of solutions of the wave equation propagate 
with time along the bicharacteristics introduced above. This is a particular instance 
of a more general fact that applies to general PDEs and can be found in [46, 84]. 
As a result, if after time T all the rays leave the domain Q of interest, the solution 
becomes smooth (infinitely differentiable) inside Q. 

The notion of so-called local energy decay, which we survey next, is important 
for the understanding of the non-trapping conditions in TAT. 


Local Energy Decay Estimates 

Assuming that the initial data f(x) (1) is compactly supported and the speed c(x) 
is non-trapping, one can provide the local energy decay estimates [32, 90, 91]. 
Namely, in any bounded domain Q, the solution p(x,t) of (1) satisfies, for a 
sufficiently large Ty and for any (k, m), the estimate 


gk tlm 


kam <CrmveOWfllnz, for x € Q,t > Tp. (11) 
t\x 


Here v(t) = 1~"*!-* for even n and v(t) = e~™ for odd n and some 5 > 0. Any 
value Tp larger than the diameter of &2 works in this estimate. 


Uniqueness Result for Non-trapping Speeds 

If the speed is non-trapping, the local energy decay allows one to start solving 
the problem (3) from t = oo, imposing zero conditions at f = oo and using 
the measured data g as the boundary conditions. This leads to recovery of the 
whole solution and in particular its initial value f(x). As the result, one obtains 
the following simple uniqueness result of [3]: 
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Theorem 7 ([3]). Ifthe speed c(x) is smooth and non-trapping and the acquisition 
surface S is closed, then the TAT data g(y,t) determines the tomogram f(x) 
uniquely. 


Notice that the statement of the theorem holds even if the support of f is not 
completely inside of the acquisition surface S. 


Uniqueness Results for Finite Observation Times 

As in the case of constant coefficients, if the speed of sound is non-trapping, 
appropriately long finite observation time suffices for the uniqueness. Let us denote 
by T(Q) the supremum of the time it takes the ray to reach S, over all rays 
originating in Q. In particular, if c(x) is trapping, T((&2) might be infinite. 


Theorem 8 ((86]). The data g measured till any time T larger than T(Q) is 
sufficient for the unique recovery of f . 


Stability 


By stability of reconstruction of the TAT tomogram f from the measured data g, 
we mean that small variations of g in an appropriate norm lead to small variations 
of the reconstructed tomogram f, also measured by an appropriate norm. In other 
words, small errors in the data lead to small errors in the reconstruction. 

We will try to give the reader a feeling of the general state of affairs with stability, 
referring to the literature (e.g., [5,48,54, 71, 86]) for further exact details. 

We will consider as functional spaces the standard Sobolev spaces H’ of 
smoothness s. We will also denote, as before, by WV the operator transforming the 
unknown ff into the data g. 

Let us recall the notions of Lipschitz and Hdélder stability. An even weaker 
logarithmic stability will not be addressed here. The reader can find discussion of 
the general stability notions and issues, as applied to inverse problems, in [49]. 


Definition 5. The operation of reconstructing f from g is said to be Lipschitz 


stable between the spaces H*? and H"', if the following estimate holds for some 
constant C: 


IIfllas < Clgllae. 


The reconstruction is said to be Hélder stable (a weaker concept), if there are 
constants 51, 52, 53,C, 4 > 0, and 6 > O such that 


If lla < Clee 


for all f such that || f ||q3 < 6. 
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Stability can be also interpreted in the terms of the singular values o; of the 
forward operator f +> g in L”, which have at most power decay when j —> oo. The 
faster is the decay, the more unstable the reconstruction becomes. The problems with 
singular values decaying faster than any power of j are considered to be extremely 
unstable. Even worse are the problems with exponential decay of singular values 
(analytic continuation or solving Cauchy problem for an elliptic operator belong to 
this class). Again, the book [49] is a good source for finding detailed discussion of 
such issues. 

Consider as an example inversion of the standard in X-ray CT and MRI Radon 
transform that integrates a function f over hyperplanes in R”. It smoothes function 
by “adding (n — 1)/2 derivatives.” Namely, it maps continuously H*-functions in 
Q into the Radon projections of class H’+~/?, Moreover, the reconstruction 
procedure is Lipschitz stable between these spaces (see [63] for detailed discussion). 

One should notice that since the forward mapping is smoothing (it “adds 
derivatives” to a function), the inversion should produce functions that are less 
smooth than the data, which is an unstable operation. The rule of thumb is that 
the stronger is the smoothing, the less stable is the inversion (this can be rigorously 
recast in the language of the decay of singular values). Thus, problems that require 
reconstructing non-smooth functions from infinitely differentiable (or even worse, 
analytic) data are extremely unstable (with super-algebraic or exponential decay of 
singular values correspondingly). This is just a consequence of the standard Sobolev 
embedding theorems (see, e.g., how this applies in TAT case in [65]). 

In the case of a constant sound speed and the acquisition surface completely 
surrounding the object, as we have mentioned before, the TAT problem can be 
recast as inversion of the spherical mean transform M (see Sect. 2). Due to analogy 
between the spheres centered on S and hyperplanes, one can conjecture that the 
Lipschitz stability of the inversion of the spherical mean operator /V/ is similar 
to that of the inversion of the Radon transform. This indeed is the case, as long 
as f is supported inside S, as has been shown in [71]. In the cases when closed- 
form inversion formulas are available (see section “Constant Speed of Sound”), this 
stability can also be extracted from them. If the support of f does reach outside, 
reconstruction of the part of f that is outside is unstable (i.e., is not even Hélder 
stable, due to the reasons explained in section “Incomplete Data’). 

In the case of variable non-trapping speed of sound c(x), integral geometry 
does not apply anymore, and one needs to address the issue using, for instance, 
time reversal. In this case, stability follows by solving the wave equation in reverse 
time starting from tf = ov, as it is done in [3]. In fact, Lipschitz stability in this 
case holds for any observation time exceeding T (2) (see [86], where microlocal 
analysis is used to prove this result). 

The bottom line is that TAT reconstruction is sufficiently stable, as long as the 
speed of sound is non-trapping. 

However, trapping speed does cause instability [48]. Indeed, since some of the 
rays are trapped inside (2, the information about some singularities never reaches 
S (no matter for how long one collects the data), and thus, as it is shown in [65], 
the reconstruction is not even Holder stable between any Sobolev spaces, and the 
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singular values have super-algebraic decay. See also section “Incomplete Data” 
below for a related discussion. 


Incomplete Data 


In the standard X-ray CT, incompleteness of data arises, for instance, if not all 
projection angles are accessible or irradiation of certain regions is avoided, or as 
in the ROI (region of interest) imaging, only the ROI is irradiated. 

It is not that clear what incomplete data means in TAT. Usually one says that one 
deals with incomplete TAT data if the acquisition surface does not surround the 
object of imaging completely. For instance, in breast imaging it is common that 
only a half-sphere arrangement of transducers is possible. We will see, however, 
that incomplete data effects in TAT can also arise due to trapping, even if the 
acquisition surface completely surrounds the object. 

The questions addressed here are the following: 


1. Is the collected incomplete data sufficient for unique reconstruction? 
2. If yes, does the incompleteness of the data have any effect on the stability and 
quality of the reconstruction? 


Uniqueness of Reconstruction 

Uniqueness of reconstruction issues can be considered essentially resolved for 
incomplete data in TAT, at least in most situations of practical interest. We will 
briefly survey here some of the available results. In what follows, the acquisition 
surface S' is not closed (otherwise the problem is considered to have complete data). 


Uniqueness for Acoustically Homogeneous Media 

In this case, Theorem 4 contains some useful sufficient conditions on S that 
guarantee uniqueness. Microlocal results of [7,61, 85], as well as the PDE approach 
of [36] further applied in [8], provide also some other conditions. We assemble some 
of these in the following theorem: 


Theorem 9. Let S be a non-closed acquisition surface in TAT. Each of the 
following conditions on S is sufficient for the uniqueness of reconstruction of any 
compactly supported function f from the TAT data collected on S: 


1. Surface S is not algebraic (i.e., there is no nonzero polynomial vanishing on S). 
2. Surface S is a uniqueness set for harmonic polynomials (i.e., there is no nonzero 
harmonic polynomial vanishing on S ). 

. Surface S contains an open piece of a closed analytic surface T. 

4. Surface S contains an open piece of an analytic surface T separating the space 
R" such that f is supported on one side of T. 

5. For some point y € S, the function f is supported on one side of the tangent 
plane Ty to S at y. 


OW 
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For instance, if the acquisition surface S is just a tiny non-algebraic piece of a 
surface, data collected on S determines the tomogram f uniquely. However, one 
realizes that such data is unlikely to be useful for any practical reconstruction. Here 
the issue of stability of reconstruction kicks in, as it will be discussed in the stability 
subsection further down. 


Uniqueness for Acoustically Inhomogeneous Media 
In the case of a variable speed of sound, there still are uniqueness theorems for 
partial data [86, 87], e.g.: 


Theorem 10 ([86]). Let S be an open part of the boundary 0Q of a strictly convex 
domain &2, and the smooth speed of sound equals | outside 02. Then the TAT data 
collected on S for atime T > T(Q) determines uniquely any function f € Hj(&), 
whose support does not reach the boundary. 


A modification of this result that does not require strict convexity is also available 
in [87]. 

While useful uniqueness of reconstruction results exists for incomplete data 
problems, all such problems are expected to show instability. This issue is discussed 
in the subsections below. This will also lead to a better understanding of incomplete 
data phenomena in TAT. 


“Visible” (“Audible”) Singularities 

According to the discussion in section “Acoustically Inhomogeneous Media,’ the 
singularities (the points of the wave front set WF (/) of the function f in (3)) are 
transported with time along the bicharacteristics (10). Thus, in the x-space they 
are transported along the geometric rays. These rays may or may not reach the 
acquisition surface S, which triggers the introduction of the following notion: 


Definition 6. A phase space point (xo, &) is said to be “visible” (sometimes the 
word “audible” is used instead), if the corresponding ray (see (10)) reaches in finite 
time the observation surface S. 

A region U C R’ is said to be in the visibility zone, if all points (xo, &)) with 
xo € U are visible. 


An example of wave propagation through inhomogeneous medium is presented 
in Fig.5. The open observation surface S in this example consists of the two 
horizontal and the left vertical sides of the square. Figure 5a shows some rays 
that bend, due to acoustic inhomogeneity, and leave through the opening of the 
observation surface S (the right side of the square). Figure 5b presents a flat 
phantom, whose wave front set creates these escaping rays, and thus is mostly 
invisible. Then Fig. 5c—f shows the propagation of the corresponding wave front. 

Since the information about the horizontal boundaries of the phantom escapes, 
one does not expect to reconstruct it well. Figure 6 shows two phantoms and 
their reconstructions from the partial data: (a—b) correspond to the vertical flat 
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Fig. 5 (a) Some rays starting along the interval x € [—0.7, —0.2] in the vertical directions escape 
on the right; (b) a flat phantom with “invisible wave front”; (c-f) propagation of the flat front: most 
of the energy of the signal leaves the square domain through the hole on the right 


phantom, whose only invisible singularities are at its ends. One sees essentially 
good reconstruction, with a little bit of blurring at the end points. On the other 
hand, reconstruction of the horizontal phantom with almost the whole wave front 
set invisible does not work. The next Fig. 7 shows a more complex square phantom, 
whose singularities corresponding to the horizontal boundaries are invisible, while 
the vertical boundaries are fine. One sees clearly that the invisible parts have 
been blurred away. On the other hand, Fig. 11a in Sect.4 shows that one can 
reconstruct an image without blurring and with correct values, if the image is 
located in the visibility region. The reconstructed image in this figure is practically 
indistinguishable from the phantom shown in Fig. 10a 


Remark 2. If S is a closed surface and x9 is a point outside of the convex hall of 
S, there is a vector & 4 0 such that (xo, &) is “invisible.” Thus, the visibility zone 
does not reach outside the closed acquisition surface S. 


Stability of Reconstruction for Incomplete Data Problems 

In all examples above, uniqueness of reconstruction held, but the images were still 
blurred. The question arises whether the blurring of “invisible” parts is avoidable 
(after all, the uniqueness theorems seem to claim that “everything is visible”). The 
answer to this is, in particular, the following result of [65], which is an analogue of 
similar statements in X-ray tomography: 
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Fig. 6 Reconstruction with the same speed of sound as in Fig. 5: (a—b) a phantom with strong 
vertical fronts and its reconstruction; (c-d) a phantom with strong horizontal fronts and its 
reconstruction 


Fig. 7 Reconstruction with the same speed of sound as in Fig.5: (a) a phantom; (b) its 
reconstruction; (c) a magnified fragment of (b) 


Theorem 11 ((65]). Jf there are invisible points (xo, &) in Q x (Rz\0), then 


inversion of the forward operator W is not Holder stable in any Sobolev spaces. 
The singular values 0; of W in L? decay super-algebraically. 
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Thus, the presence of invisible singularities makes the reconstruction severely 
ill-posed. In particular, according to Remark 2, this theorem implies the following 
statement: 


Corollary 1. Reconstruction of the parts of f(x) supported outside the closed 
observation surface S is unstable. 


On the other hand, 


Theorem 12 ((86]). All visible singularities of f can be reconstructed with Lips- 
chitz stability (in appropriate spaces). 


Such a reconstruction of visible singularities can be obtained in many ways, 
for instance, just by replacing the missing data by zeros (with some smoothing 
along the junctions with the known data, in order to avoid artifact singularities). 
However, there is no hope for stable recovery of the correct values of f(x), if there 
are invisible singularities. 


Discussion of the Visibility Condition 


Visibility for Acoustically Homogeneous Media 
In the constant speed case, the rays are straight, and thus the visibility condition has 
a simple test: 


Proposition 1 (e.g., [48, 99, 100]). If the speed is constant, a point xo is in the 
visible region, if and only if any line passing through Xo intersects at least once 
the acquisition surface S (and thus a detector location). 


Figure 8 illustrates this statement. It shows a square phantom and its reconstruc- 
tion from complete data and from the data collected on the half circle S surrounding 
the left half of object. The parts of the interfaces where the normal to the interface 
that does not cross S are blurred. 


Visibility for Acoustically Inhomogeneous Media 
When the speed of sound is variable, an analogue of Proposition | holds, with lines 
replaced by rays. 


Proposition 2 (e.g., [48, 65, 86]). A point xo is in the visible region, if and only 
if for any & 4 0 at least one of the two geometric rays starting at (xo, &) and 
at (xo, —&) (see (10)) intersects the acquisition surface S (and thus a detector 
location). 


The reader can now see an important difference between the acoustically 
homogeneous and inhomogeneous media. Indeed, even if S surrounds the support 
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Fig. 8 Reconstruction from incomplete data using closed-form inversion formula in 2D; detectors 
are located on the left half circle of radius 1.05: (a) phantom; (b) reconstruction from complete 
data; and (c) reconstruction from the incomplete data 


of f completely, trapped rays will never find their way to S', which will lead, as we 
know by now, to instabilities and blurring of some interfaces. 

Thus, the presence of rays trapped inside the acquisition surface creates effects 
of incomplete data type. This is exemplified in Fig. 9 with a square phantom and its 
reconstruction shown in the presence of a trapping (parabolic) speed. Notice that the 
square centered at the center of symmetry of the speed is reconstructed very well 
(see Fig. 9d), since none of the rays carrying its singularities is trapped. 


Range Conditions 


In this section we address the problem of describing the ranges of the forward 
operators W (see (4)) and M (see (8)), the latter in the case of an acoustically 
homogeneous medium (i.e., for c = const). The ranges of these operators, 
similarly to the range of the Radon and X-ray transforms (see [63]), are of 
infinite co-dimensions. This means that ideal data g from a suitable function space 
satisfy infinitely many mandatory identities. Knowing the range is useful for many 
theoretical and practical purposes in various types of tomography (reconstruction 
algorithms, error corrections, incomplete data completion, etc.), and thus this topic 
has attracted a lot of attention (e.g., [41,45, 63, 68, 70] and references therein). 

As we will see in the next section, range descriptions in TAT are also intimately 
related to the recovery of the unknown speed of sound. 

We recall [41, 45, 63] that for the standard Radon transform 


fe) > g(s,o) = / Sasi at. 


where f is assumed to be smooth and supported in the unit ball B = {x | |x| < 1}, 
the range conditions on g(s, @) are: 
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Fig. 9 Reconstruction of a square phantom from full data in the presence of a trapping parabolic 
speed of sound (the speed is radial with respect to the center of the picture): (a) an off-center 
phantom; (b) its reconstruction; (c) a magnified fragment of (b); (d) reconstruction of a centered 
square phantom 


1. Smoothness and support: g € Cf? ([-1, 1] x S), where S is the unit sphere of 
vectors w 

2. Evenness: g(—s, —@) = g(s,@) 

3. Moment conditions: for any integer k > 0, the kth moment 


fo) 


G;(@) = [ s:e(o.s)as 


—oo 


extends from the unit sphere S to a homogeneous polynomial of degree k in w. 


The seemingly “trivial” evenness condition is sometimes the hardest to generalize 
to other transforms of Radon type, while it is often easier to find analogues of the 
moment conditions. This is exactly what happens in TAT. 
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For the operators WV, M in TAT, some sets of range conditions of the moment 
type had been discovered over the years [7, 60,78], but complete range descriptions 
started to emerge only since 2006 [2, 4-6, 9, 37, 54]. 

Range descriptions for the more general operator WV are harder to obtain than for 
M, and complete range descriptions are not known for even dimensions or for the 
case of the variable speed of sound. 

Let us address the case of the spherical mean operator M first. 


The Range of the Spherical Mean Operator M 

The support and smoothness conditions are not hard to come up with, at least when 
S is a sphere. By choosing an appropriate length scale, we can assume that the 
sphere is of radius | and centered at the origin and that the interior domain Q is 
the unit ball B = {x | |x| = 1}. If f is smooth and supported inside B (i.e., 
Sf € C§°(B)), then it is clear that the measured data satisfies the following: 


Smoothness and support conditions: 
g €Co(S x [0, 2)). (12) 


An analogue of the moment conditions for g(y,r) := Mf was implicitly 
present in [7,60] and explicitly formulated as such in [78]: 


Moment conditions: for any integer k > 0, the moment 


oe) 


Mi (y) = / kta! oy r)dr (13) 
0 


extends from S to a (in general, nonhomogeneous) polynomial Q,(x) of degree 
at most 2k. 

These two types of conditions happen to be incomplete, i.e., infinitely many 
others exist. The Radon transform experience suggests to look for an analogue of 
evenness conditions. And indeed, a set of conditions called orthogonality conditions 
was found in [5,9, 37]. 

Orthogonality conditions: Let —At be the eigenvalue of the Laplace operator A 
in B with zero Dirichlet conditions and wx be the corresponding eigenfunctions. 
Then the following orthogonality condition is satisfied: 


8(x, 1) 0 Va(X) inant" dxdt = 0. (14) 
Sx[0,2] 


Here jp(Z) = €pz ? Jp(z) is the so-called spherical Bessel function. 

The range descriptions obtained in [5, 9,37] demonstrated that these three types 
of conditions completely describe the range of the operator M on functions f € 
Cy? (B). At the same time, the results of [5,37] showed that the moment conditions 
can be dropped in odd dimensions. It was then discovered in [2] that the moment 
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conditions can be dropped altogether in any dimension, since they follow from the 
other two types of conditions: 


Theorem 13 ([2]). Let S be the unit sphere. A function g(y,t) on the cylinder S x 
R* can be represented as Mf for some f € C&°(B) if an only if it satisfied the 
above smoothness and support and orthogonality conditions (12), (14). 

The statement also holds in the finite smoothness case, if one replaces the 
requirements by f € Hj(B) and g € peer(s x [0, 2]). 


The range of the forward operator M has not been described when S is not a 
sphere, but, say, a convex smooth closed surface. The moment and orthogonality 
conditions hold for any S', and appropriate smoothness and support conditions can 
also be formulated, at least in the convex case. However, it has not been proven that 
they provide the complete range description. 

It is quite possible that for nonspherical S the moment conditions might have to 
be included into the range description. 

A different range description of the Fredholm alternative type was developed in 
[71] (see also [39] for description of this result). 


The Range of the Forward Operator W 

We recall that the operator W (see (4)) transforms the initial value f in (3) into 
the observed S values g of the solution. There exist Kirchhoff—Poisson formulas 
representing the solution p and thus g = Wf in terms of the spherical means of 
f (e., in terms of M f). However, translating the result of Theorem 13 into the 
language of WV is not straightforward, since in even dimensions these formulas are 
nonlocal ([27] p. 682): 


i) (n—3)/2 
Why.) = a (75) t" (Mf) (y,t), for odd n. (15) 
and 
1 a\° 2? Fr (MS) 0.7) 
Wf(y,t) = T(n/2) (5) Jz dr, forevenn. (16) 


The nonlocality of the transformation for even dimensions reflects the absence of 
Huygens’ principle (i.e., absence of sharp rear fronts of waves) in these dimensions; 
it also causes difficulties in establishing the complete range descriptions. In 
particular, due to the integration in (16), M f(y, t) does not vanish for large times 
t anymore. One can try to use other known operators intertwining the two problems 
(see [5] and references therein), and some of which do preserve vanishing for large 
values of ¢, but this so far has led only to very clumsy range descriptions. 

However, for odd dimensions, the range description of WV can be obtained. In 
order to do so, given the TAT data g(y, r), let us introduce an auxiliary time-reversed 
problem in the cylinder B x [0, 2]: 
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du — Aq = 0 for (x,t) € B x [0, 2]), 
q(x,2) = q:(x,2) = Oforx € B, (17) 
q(y.t) = g(y,t) for (y,t) € S x [0,2]). 


We can now formulate the range description from [37, 39]: 


Theorem 14 ([37, 39]). For odd dimensions n and S being the unit sphere, a 
function g € Cy°(S x [0,2]) can be represented as Wf for some f € Cy°(B) 
if and only if the following condition is satisfied: 


The solution q of (17) satisfies q:(x,0) = O forall x € B. 


Orthogonality type and Fredholm alternative type range conditions, equivalent to 
the one in the theorem above, are also provided in [37,39]. 


Reconstruction of the Speed of Sound 


Unsurprisingly, all inversion procedures outlined in Sect. 4 rely upon the knowledge 
of the speed of sound c(x). Although often, e.g., in breast imaging, the medium is 
assumed to be acoustically homogeneous, this is not a good assumption in many 
other cases. It has been observed (e.g., [48,50]) that replacing even slightly varying 
speed of sound with its average value might significantly distort the image; not 
only the numerical values, but also the shapes of interfaces between the tissues 
will be reconstructed incorrectly. Thus, the question of estimating c(x) correctly 
becomes important. One possible approach [50] is to use an additional transmission 
ultrasound scan to reconstruct the speed beforehand. The question arises of whether 
one could determine the speed of sound c(x) and the tomogram f(x) (assuming 
that f is not zero) simultaneously from the TAT data. In fact, one needs only to 
determine c(x) (without knowing f), since then inversion procedures of Sect.4 
would apply to recover f. 

At the first glance, this seems to be an overly ambitious project. Indeed, if we 
denote the forward operator WV by YW, to indicate its dependence on the speed of 
sound c(x), then the problem becomes, given the data g, to find both c and f from 
the equality 


W.f = &. (18) 


A similar situation arises in the SPECT emission tomography (see [63] and 
references therein), where the role of the speed of sound is played by the unknown 
attenuation. It is known, however, that in SPECT the attenuation can be recovered 
for a “generic” f. 

What is the reason for such a strange situation? It looks like for any c one could 
solve Eq. (18) for an f, and thus no information about c is contained in the data 
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g. This argument is incorrect for the following reason: the range of the forward 
operator, as we know already from the previous section, has infinite co-dimension. 
Thus, this range has a lot of space to “rotate” when c changes. Imagine for an 
instance that the rotation is so powerful that for different values of c the ranges 
have only zero (the origin) in common. Then, knowing g in the range, one would 
know which c it came from. Thus, the problem of recovering the speed of sound 
from the TAT data is closely related to the range descriptions. 

Numerical inversions using algebraic iterative techniques (e.g., [102, 104]) show 
that recovering both c and f might be indeed possible. 

Unfortunately, very little is known at the moment concerning this problem. 
Direct usage of range conditions attempted in [48] has led only to extremely weak 
and not practically useful results so far. A revealing relation to the transmission 
eigenvalue problem well known in inverse problems (see [26] for the survey) 
was recently discovered by D. Finch. Unfortunately, the transmission eigenvalue 
problem remains still unresolved. However, one can derive from this relation the 
following result regarding uniqueness of the reconstruction of the speed of sound, 
due to M. Agranovsky (a somewhat restricted version is due to D. Finch et al., both 
unpublished): 


Theorem 15. /f two speeds satisfy the inequality c\(x) = c2(x) for all x € Q and 
produce for some functions f, f2 the same nonzero TAT data g (i.e, Wa fi = 
g,We2 fo = g), then c,(x) = c2(x). 


It is known [49, Corollary 8.2.3] that if a function f(x) is such that A f(x) 4 0 
and for two acoustic speeds c;(x) and c2(x), it produces the same TAT data g, then 
Cj = C2. 

It is clear that the problem of finding the speed of sound from the TAT data is 
still mostly unresolved. 


4 Reconstruction Formulas, Numerical Methods, 
and Case Examples 


Numerous formulas, algorithms, and procedures for reconstruction of images from 
TAT measurements have been developed by now. Most of these techniques require 
the data being collected on a closed surface (closed curve in 2D) surrounding the 
object to be imaged. Such methods are discussed in section “Full Data (Closed 
Acquisition Surfaces).” We review methods that work under the assumption of 
constant speed of sound in section “Constant Speed of Sound.” The techniques 
applicable in the case of the known variable speed of sound are considered in 
section “Constant Speed of Sound.” Closed surface measurements cannot always 
be implemented, since in some practical situations the object cannot be completely 
surrounded by the detectors. In this case, one has to resort to various approximate 
reconstruction techniques as discussed in section “Partial (Incomplete) Data.” 
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Full Data (Closed Acquisition Surfaces) 


Constant Speed of Sound 

When the speed of sound within the tissues is a known constant, the TAT problem 
can be reformulated (see Sect. 2) in terms of the values of the spherical means of the 
initial condition f(x). These means can be easily recovered from the measurements 
of the acoustic pressure using formulas (15) and (16) (see the discussion in [7]). In 
this case, image reconstruction becomes equivalent to inverting the spherical mean 
transform M. Thus, in what follows, we consider the problem of reconstructing 
a function f(x) supported within the region bounded by a closed surface S from 
known values of its spherical integrals g(y,r) with centers on S 


g(.r) = / fy +ro)rdo, y€S, (19) 
gi-l 


where dw is the standard measure on the unit sphere. 


Series Solutions for Spherical Geometry 

The first inversion procedures for the case of closed acquisition surfaces were 
described in [66,67], where solutions were found for the cases of circular (in 2D) 
and spherical (in 3D) surfaces, respectively. These solutions were obtained by the 
harmonic decomposition of the measured data and of the sought function f(x), 
followed by equating coefficients of the corresponding Fourier series. In particular, 
the 2D algorithm of [66] pertains to the case when the detectors are located on a 
circle of radius R. This method is based on the Fourier decomposition of f and g 
in angular variables 


fx) = D0 f(pe"*?, x = (pcos(g), psin(y)) (20) 
g(v().r) = > ge(r)el*’, y = (Reos(4), Rsin(6)), 


where 
(Hu) (s) = 20 ye u(t) Jin (st)tdt 
0 


is the Hankel transform and J,,(t) is the Bessel function. As shown in [66], the 
Fourier coefficients f;,(o) can be recovered from the known coefficients g,(r) by 
the following formula: 


_ 1 g(r) 
he) = 4m (Saray [Ser |): 
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This method requires division of the Hankel transform of the measured data by 
the Bessel functions J;, which have infinitely many zeros. Theoretically, there is 
no problem: the range conditions (section “Range Conditions’) on the exact data g 
imply that the Hankel transform Ho [ (2x r) 85 (r)] has zeros that cancel those in 
the denominator. However, since the measured data always contain errors, the exact 
cancelation does not happen, and one needs a sophisticated regularization scheme 
to guarantee that the error remains bounded. 

This difficulty can be avoided (see, e.g., [54]) by replacing the Bessel function Jo 
in the inner Hankel transform by the Hankel function HS”. This yields the following 
formula for fi, (p) : 


Sx(p) = Hx ( esi tannr) ' 


Teh 
Ay? ALR) Jo 


Unlike J,,, Hankel functions AO) do not have zeros for any real values of 
t, which removes the problems with division by zeros [66]. (A different way of 
avoiding divisions by zero was found in [44].) 

This derivation can be repeated in 3D, with the exponentials e'"” replaced by 
the spherical harmonics and with cylindrical Bessel functions replaced by their 
spherical counterparts. By doing this, one arrives at the Fourier series method of 
[67] (see also [97]). The use of the Hankel function Hy above is similar to the way 


iko 


the spherical Hankel function Ae is utilized in [67] to avoid the divisions by zero. 


Eigenfunction Expansions for a General Geometry 
The series methods described in the previous section rely on the separation of 
variables that occurs only in spherical geometry. A different approach was proposed 
in [58]. It works for arbitrary closed surfaces, but is practical only for those 
with explicitly known eigenvalues and eigenfunctions of the Dirichlet Laplacian in 
the interior. These include, in particular, the surfaces of such bodies as spheres, 
half spheres, cylinders, cubes, and parallelepipeds, as well as the surfaces of 
crystallographic domains. 

Let A and u), (x) be the eigenvalues and an orthonormal basis of eigenfunctions 
of the Dirichlet Laplacian —A in the interior Q of a closed surface S: 


Aum(x) + A2,um(x) =0, xEQ, QCR’, (21) 
Un(x) = 0, x eS, 
inl =f luna = 1, 
Q 
As before, one would like to reconstruct a compactly supported function f(x) from 


the known values of its spherical integrals g(y, 1) (see (19)) with centers on S. Since 
Um(X) is the solution of the Dirichlet problem for the Helmholtz equation with zero 
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boundary conditions and the wave number (,,, this function admits the Helmholtz 
representation 


0 
tin (2) = [ Sin (lx — YD e—tim(ydds(y) x € (22) 


where ®,,, (|x — y|) is a free-space Green’s function of the Helmholtz equation (21) 
and n is the exterior normal to S. 
The function f(x) can be expanded into the series 


f(x) = ~ Q@mUm(x), where (23) 


m=0 


An = [unto Fea. 


A reconstruction formula for a, (and thus for f(x)) will result, if one substitutes 
representation (22) into (23) and interchanges the orders of integration 


a 
Om = | un(s floras = [10 dn) stm 9A) (24) 


where 


diam Q 
ipay= i &, (|x — yl) f@)dx = / ely. r)®i(r)dr. (25) 


Now f(x) can be obtained by summing the series (23). This method becomes 
computationally efficient when the eigenvalues and eigenfunctions are known 
explicitly, especially if a fast summation formula for the series (23) is available. 
This is the case when the acquisition surface S is the surface of a cube, and thus 
the eigenfunctions are products of sine functions. The resulting 3D reconstruction 
algorithm is extremely fast and precise (see [58]). 

The above method has an interesting property. If the support of the source f(x) 
extends outside Q, the algorithm still yields theoretically an exact reconstruction of 
F(x) inside Q. Indeed, the value of the expression (22) for all x lying outside Q is 
zero. Thus, when one computes (24) for x € R"\Q, values of f(x) are multiplied 
by zero and do not affect further computation in any way. This feature is shared 
by the time-reversal method (see the corresponding paragraph in section “Constant 
Speed of Sound”). The closed-form FBP-type reconstruction techniques considered 
in the next subsection do not have this property. In other words, in the presence of a 
source outside the measurement surface, reconstruction within Q can be incorrect. 

The reason for this difference is that all currently known closed-form FBP-type 
formulas rely (implicitly or explicitly) on the assumption that the wave propagates 
outside S in the whole free space and has no sources outside. On the other hand, 
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the eigenfunction expansion method and the time reversal rely only upon the time 
decay of the wave inside S, which is not influenced by f having a part outside S. 


Closed-Form Inversion Formulas 

Closed-form inversion formulas play a special role in tomography. They bring 
about better theoretical understanding of the problem and frequently serve as 
starting points for the development of efficient reconstruction algorithms. A well- 
known example of the use of explicit inversion formulas is the so-called filtered 
backprojection (FBP) algorithm in X-ray tomography, which is derived from one of 
the inversion formulas for the classical Radon transform (see, e.g., [63]). 

The very existence of closed-form inversion formulas for TAT had been in doubt, 
till the first such formulas were obtained in odd dimensions by Finch et al. in [36], 
under the assumption that the acquisition surface S is a sphere. Suppose that the 
function f(x) is supported within a ball of radius R and that the detectors are located 
on the surface S = 0B of this ball. Then some of the formulas obtained in [36] read 
as follows: 


“ti g(y. ly — x) 
fe) =-zaghs | aay, (26) 
1 1 @ 
fey =-saq [ (Fzae0-)| aon, @n 
8 r=|y—x| 
= 1 1a 0 g(y,r) 
Ff) = ~ 872R i (; or (« or or )) dA(y), Ce) 
dB 


rly 


where dA(y) is the surface measure on 0B and g represents the values of the 
spherical integrals (19). 

These formulas have an FBP (filtered backprojection) nature. Indeed, differenti- 
ation with respect to r in (27) and (28) and the Laplace operator in (26) represent 
the filtration, while the (weighted) integrals correspond to the backprojection, 1.e., 
integration over the set of spheres passing through the point of interest x and 
centered on S. 

The so-called universal backprojection formula in 3D was found in [98] (it is 
also valid for the cylindrical and plane acquisition surfaces, see section “Partial 
(Incomplete) Data’). In our notation, this formula takes the form 


: i) dA(y), (29) 


Ll .: 
fs) = gaan fi (25 


0B = - 
r=ly—x| 


1150 P. Kuchment and L. Kunyansky 


or, equivalently, 


a: 0d (10 gir) 
FQ) = ol an (- or or ) dA(y), (30) 
aB 


r=|y—x| 


where n(y) is the exterior normal vector to 0B. One can show [4, 64, 98] that 
formulas (26) through (29) are not equivalent on non-perfect data: the result will 
differ if these formulas are applied to a function that does not belong to the range 
of the spherical mean transform M. A family of inversion formulas valid in R” for 
arbitrary n > 2 was found in [57] 


1 
f) = Faardiv | mOYHLy. |e y)dAO, GI) 
aB 
where 
2R 2R 

h(y,t) = [veo [ 1@ns0.ndr—Jan f YAnso.ndr A"-3.d X32) 

Rt+ 0 0 

Jn /2— Yn /2— 

Oe 3) 


and J,,/2-1(t) and Y,,/2-1(t) are, respectively, the Bessel and Neumann functions of 
order n/2—1.In 3D, J(t) and Y(t) are simply ¢~! sint and t~! cosf, and formulas 
(31) and (32) reduce to (30). 

In 2D, Eq. (32) also can be simplified [4], which results in the formula 


2R 
1. 1 
f(x) = scativ [ ny) [soos per dl(y), (34) 
aB 0 


where 0B now stands for the circle of radius R and d/(y) is the standard arc length. 

A different set of closed-form inversion formulas applicable in even dimensions 
was found in [35]. Formula (34) can be compared to the following inversion 
formulas from [35] 


2R 
1 

fix) = 9 ff ely.r)oger? = |x yP) dr dig, 35) 

mR 

dB 0 
or 
i fray .d 26%) 

feos ff ge (rE) ose? -be- Py araiyy. G6) 

dB O 
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Finally, a unified family of inversion formulas was derived in [64]. In our 
notation, it has the following form 


4 a 5 
fey =- f (5-&0.9) ay 446), CO 
mR ot |x — y| 
ua t=|x-y| 
1 
K,(y,0) = rays | PVN [ sango.nar dh 
Rt+ Rt+ 


where OB is the surface of a ball in R” of radius R, functions J and Y are in (33), 
and & is an arbitrary fixed vector. In particular, in 3D 


2 sint 2 t 
10) = {2 3 = (2 
mz t x tft 


and, after simple calculation, the above inversion formula reduces to 


f(x) = 


1 IG i) ae) SPIO ET gyi. 


~ 872R orror r Ix —y| 
aB aig 

r=|x—y| 
(38) 
Different choices of vector € in the above formula result in different inversion 
formulas. For example, if £ is set to zero, the ratio “ye equals R cosa, where 
a is the angle between the exterior normal n(y) and the vector y—x; when combined 
with the derivative in ft, this factor produces the normal derivative, and the inversion 
formula (38) reduces to (30). On the other hand, the choice of € = x in (38) leads 


to a formula 


1 010 g(r) 
fla) =a f(r ) dA(y), 
0B 


orror r 


r=|x—y| 
which is reminiscent of formulas (26)—(28). 


Green’s Formula Approach and Some Symmetry Considerations 

Let us suppose for a moment that the acoustic detectors could measure not only 
the pressure p(y,f) at each point of the acquisition surface S, but also the normal 
derivative dp/dn on S. Then the problem of reconstructing the initial pressure f(x) 
becomes rather simple. Indeed, one can use the knowledge of the free-space Green’s 
function for the wave equation and invoke the Green’s theorem to represent the 
solution p(x,t) of (3) in the form of integrals over S involving p(x,t) and its 


1152 P. Kuchment and L. Kunyansky 


normal derivative and the Green’s function and its normal derivative. (This can be 
done in the Fourier or time domains.) This would require infinite observation time, 
but in 3D the time T(Q) will suffice, after which the wave escapes the region of 
interest (a cutoff also would work approximately in 2D similar to the time-reversal 
method). This Green’s function approach happens to be, explicitly or implicitly, the 
starting point of all closed-form inversions described above. The trick is to rewrite 
the formula in such a way that the unknown in reality normal derivative dp/dn 
disappears from the formula. 

This was achieved in [57] by reducing the question to some integrals involving 
special functions and making the key observation that the integral 


) 
hoy) = f JAlx-deVAly-ada@, x.y e BCR" 
0B 


is a symmetric function of its arguments: 
h(x, y) = (y,x) forx,y¢ BCR.” (39) 


Similarly, the derivation of (37) in [64] employs the symmetry of the integral 


Ki39) = : JAlx—2)¥Aly —2)da@, rye BCR" 
0B 


In fact, the symmetry holds for any integral 


Wax. y) = / U(Alx —z)VAly —z)dA@, x.y © BCR’, 
OB 


where U(A|x|) and V(A|x|) are any two radial solutions of the Helmholtz equation 
Au(x) + A7u(x) = 0. (40) 


It is straightforward to verify this symmetry when S is a sphere and B is the 
corresponding ball, and the points x, y lie on the boundary S' only, rather than 
anywhere in B. This follows immediately from the rotational symmetry of S. The 
same is true for the normal derivatives on S of W,(x, y) in x and y. 

This boundary symmetry happens to imply the needed full symmetry (39) 
forx,ye B. 

Indeed, W(x, y) is a solution of the Helmholtz equation separately as a function 
of x and of y. Let us introduce a family of solutions {w, (x)}°2., of (40) in B, such 
that the members of this family form an orthonormal basis for all solutions of the 
latter equation in B. For example, the spherical waves, i.e., the products of spherical 
harmonics and Bessel functions, can serve as such a basis. 
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Then W,(x, y) can be expanded n the following series: 


Wilx.¥) = Y2 SS BamWn(V)Wn (x). (41) 


n=0 m=0 


Since W,(x, y) is a solution to the Helmholtz equation in 0B x OB, coefficients 
bn.m are completely determined by the boundary values of W,. Since the boundary 
values are symmetric, the coefficients are symmetric, i.e., bym = bm» which by 
(41) immediately implies Wy (x, y) = Wi (y, x) for all pairs (x, y) € B x B. 

This consideration extends to infinite cylinders and planes. This explains why 
the “universal backprojection formula” (30) is valid also for infinite cylinders and 
planes [98]. Since the sort of symmetry used is shared only by these three smooth 
surfaces, we believe it is unlikely that a closed-form formula could exist for any 
other smooth acquisition surface. However, an exact formula has recently been 
obtained by L. Kunyansky for the case when observation surface S is a surface 
of a cube (unpublished). 


Algebraic Iterative Algorithms 

Iterative algebraic techniques are among the favorite tomographic methods of 
reconstruction and have been used in CT for quite a while [63]. They amount 
to discretizing the equation relating the measured data with the unknown source, 
followed by iterative solution of the resulting linear system. Iterative algebraic 
reconstruction algorithms frequently produce better images than those obtained by 
other methods. However, they are notoriously slow. In TAT, they have been used 
successfully for reconstructions with partial data [14, 15, 76], see section “Partial 
(Incomplete) Data.” 


Parametrix Approaches 

Some of the earlier non-iterative reconstruction techniques [53] were of approxi- 
mate nature. For example, by approximating the integration spheres by their tangent 
planes at the point of reconstruction and by applying one of the known inversion 
formulas for the classical Radon transform, one can reconstruct an approximation 
to the image. Due to the evenness symmetry in the classical Radon projections 
(see section “Range Conditions”), the normals to the integration planes need only 
fill a half of a unit sphere, in order to make possible the reconstruction from an 
open measurement surface. A more sophisticated approach is represented by the 
so-called straightening methods [81, 82] based on the approximate reconstruction 
of the classical Radon projections from the values of the spherical mean transform 
Mf of the function f(x) in question. These methods yield not a true inversion, 
but rather what is called in microlocal analysis a parametrix. Application of a 
parametrix reproduces the function f with an additional, smoother term. In other 
words, the locations (and often the sizes) of jumps across sharp material interfaces, 
as well as the whole wave front set WF (/), are reconstructed correctly, while the 
accuracy of the lower spatial frequencies cannot be guaranteed. (Sometimes, the 
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reconstructed function has a more general form Af, where A is an elliptic pseudo- 
differential operator [46, 84] of order zero. In this case, the sizes of the jumps 
across the interfaces might be altered.) Unlike the approximations resulting from the 
discretization of the exact inversion formulas (in the situations when such formulas 
are known), the parametrix approximations do not converge, when the discretization 
of the data is refined and the noise is eliminated. Parametrix reconstructions can 
be either accepted as approximate images or used as starting points for iterative 
algorithms. 

These methods are closely related to the general scheme proposed in [20] for 
the inversion of the generalized Radon transform with integration over curved 
manifolds. It reduces the problem to a Fredholm integral equation of the second 
kind, which is well suited for numerical solution. Such an approach amounts to 
using a parametrix method as an efficient preconditioner for an iterative solver; 
the convergence of such iterations is much faster than that of algebraic iterative 
methods. 


Numerical Implementation and Computational Examples 

By discretizing exact formulas presented above, one can easily develop accurate and 
efficient reconstruction algorithms. The 3D case is especially simple: computation 
of derivatives in the formulas (26)—(30) and (38) can be easily done, for instance, 
by using finite differences; it is followed by the backprojection (described by the 
integral over 0B), which requires prescribing quadrature weights for quadrature 
nodes that coincide with the positions of the detectors. The backprojection step is 
stable; the differentiation is a mildly unstable operation. The sensitivity to noise 
in measurements across the formulas presented above seems to be roughly the 
same. It is very similar to that of the widely used FBP algorithm of classical X-ray 
tomography [63]. In 2D, the implementation is just a little bit harder: the filtration 
step in formulas (34)—(36) can be reduced to computing two Hilbert transforms (see 
[54]), which, in turn, can be easily done in the frequency domain. 

The number of floating point operations (flops) required by such algorithms is 
determined by the slower backprojection step. In 3 D, if the number of detectors is 
m? and the size of the reconstruction grid is m x m x m, the backprojection step 
(and the whole algorithm) will require O(m>) flops. In practical terms, this amounts 
to several hours of computations on a single processor computer for a grid of size 
129 x 129 x 129. 

In 2D, the operation count is just O(m?). As it is discussed in section “Variations 
on the Theme: Planar, Linear, and Circular Integrating Detectors,” the 2D problem 
needs to be solved, when integrating line detectors are used. In this situation, the 
2D problem needs to be solved m times in order to reconstruct the image, which 
raises the total operation count to O(m*) flops. 

Figure 10 shows three examples of simulated reconstruction using formula (34). 
The phantom we use (Fig. 10a) is a linear combination of several characteristic 
functions of disks and ellipses. Figure 10b illustrates the image reconstruction 
within the unit circle from 257 equi-spaced projections each containing 129 
spherical integrals. The detectors were placed on the concentric circle of radius 1.05. 


Mathematics of Photoacoustic and Thermoacoustic Tomography 1155 


a b 


a CO 


Fig. 10 Example of a reconstruction using formula (34): (a) phantom; (b) reconstruction 
from accurate data; (c) reconstruction from the data contaminated with 15 % noise; (d) recon- 
struction from the noisy data with additional smoothing 


The image shown in Fig. 10c corresponds to the reconstruction from the simulated 
noisy data that were obtained by adding to projections values of a random variable 
scaled so that the L? intensity of the noise was 15 % of the intensity of the signal. 
Finally, Fig. 10d shows how the application of a smoothing filter (in the frequency 
domain) suppresses the noise; it also somewhat blurs the edges in the image. 


Variable Speed of Sound 
The reconstruction formulas and algorithms described in the previous section work 
under the assumption that the speed of sound within the region of interest is constant 
(or at least close to a constant). This assumption, however, is not always realistic, 
e.g., if the region of interest contains both soft tissues and bones, the speed of 
sound will vary significantly. Experiments with numerical and physical phantoms 
show [48, 50] that if acoustic inhomogeneities are not taken into account, the 
reconstructed image might be severely distorted. Not only the numerical values 
could be reconstructed incorrectly, but so would the material interface locations and 
discontinuity magnitudes. 

Below we review some of the reconstruction methods that work in acoustically 
inhomogeneous media. We will assume that the speed of sound c(x) is known, 
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smooth, positive, constant for large x, and non-trapping. In practice, a transmission 
ultrasound scan can be used to reconstruct c(x) prior to thermoacoustic reconstruc- 
tion, as it is done in [50]. 


Time Reversal 

Let us assume temporarily that the speed of sound c is constant and the spatial 
dimension is odd. Then Huygens’ principle guarantees that the sound wave will 
leave the region of interest Q in time T = c/(diam Q), so that p(x,t) = 0 for all 
x € Q andt > T. Now one can solve the wave equation back in time from t = T 
to t = 0 in the domain Q x [7, 0], with zero initial conditions at T and boundary 
conditions on S provided by the data g collected by the detectors. Then the value of 
the solution at t = 0 will coincide with the initial condition f(x) that one seeks to 
reconstruct. Such a solution of the wave equation is easily obtained numerically 
by finite difference techniques [42,48]. The required number of floating point 
operations is actually lower than that of methods based on discretized inversion 
formulas (O(m*) for time reversal on a grid m x m x m in 3D versus O(m») for 
inversion formulas), which makes this method quite competitive even in the case of 
constant speed of sound. 

Most importantly, however, the method is also applicable if the speed of sound 
c(x) is variable and/or the spatial dimension is even. In these cases, the Huygens’ 
principle does not hold, and thus the solution to the direct problem will not vanish 
within 0Q in finite time. However, the solution inside Q will decay with time. Under 
the non-trapping condition, as it is shown in (11) (see [32,90,91]), the time decay is 
exponential in odd dimensions, but only algebraic in even dimensions. Although, in 
order to obtain theoretically exact reconstruction, one would have to start the time 
reversal at 7 = oo, and numerical experiments (e.g., [48]) and theoretical estimates 
[47] show that in practice it is sufficient to start at the values of T when the signal 
becomes small enough and to approximate the unknown value of p(x, 7) by zero 
(a more sophisticated cutoff is used in [86]). This works [42,48] even in 2D (where 
decay is the slowest) and in inhomogeneous media. However, when trapping occurs, 
the “invisible” parts blur away (see section “Incomplete Data” for the discussion). 


Eigenfunction Expansions 

An “inversion formula” that reconstructs the initial value f(x) of the solution of 
the wave equation from values on the measuring surface S can be easily obtained 
using time reversal and Duhamel’s principle [3]. Consider in Q the operator A = 
—c?(x)A with zero Dirichlet conditions on the boundary S = 0Q. This operator 
is self-adjoint, if considered in the weighted space L?(Q;c~?(x)). Let us denote 
by E the operator of harmonic extension, which transforms a function ¢ on S to a 
harmonic function on Q that coincides with @ on S. Then f can be reconstructed 
[3] from the data g in (3) by the following formula: 


co 


f(x) =(Eshao)— f Atsin(cAS)E@inGadde, 42) 


0 
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which is valid under the non-trapping condition on c(x). However, due to the 
involvement of functions of the operator A, it is not clear how useful this formula 
can be. 

One natural way to try to implement numerically the formula (42) is to use the 
eigenfunction expansion of the operator A in Q (assuming that such expansion is 
known). This quickly leads to the following procedure [3]. The function f(x) can 
be reconstructed inside Q from the data g in (3), as the following L?(B)-convergent 
series: 


fx) = Vo fede), (43) 


k 


where the Fourier coefficients f, can be recovered from the data using one of the 
following formulas: 


Se = A? 8K (0) — AQ? ef sin Axt)gy (dt, 


fie = A? ge (0) + Az? i cos (Axt)g;,(t)dt, or 


fe = hz (aa (Agt)gx(t)dt = —Az ff sin (Act)g (x,t) F* Wk (x)dxdt, 
0 


(44) 
where 


= fe (x. Me Goa. 


S 


One notices that this is a generalization of the expansion method of [58] 
discussed in section “Eigenfunction Expansions for a General Geometry” to the 
case of a variable speed of sound. Unlike the algorithm of [58], this method does 
not require the knowledge of the whole space Green’s function for A (which is 
in this case unknown). However, computation of a large set of eigenfunctions 
and eigenvalues followed by the summation of the series (43) at the nodes of the 
computational grid may prove to be too time-consuming. 

It is worthwhile to mention again that the non-trapping condition is crucial for the 
stability of any TAT reconstruction method in acoustically inhomogeneous media. 
As it was discussed in section “Discussion of the Visibility Condition,” trapping can 
significantly reduce the quality of reconstruction. It is, however, most probable that 
trapping does not occur much in biological objects. 


Partial (Incomplete) Data 


Reconstruction formulas and algorithms of the previous sections work under the 
assumption that the acoustic signal is measured by detectors covering a closed 
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surface S that surrounds completely the object of interest. However, in many 
practical applications of TAT, detectors can be placed only on a certain part of 
the surrounding surface. Such is the case, e.g., when TAT is used for breast 
screening — one of the most promising applications of this modality. Thus, one 
needs methods and algorithms capable of accurate reconstruction of images from 
partial (incomplete) data, i.e., from the measurements made on open surfaces (or 
open curves in 2D). 

Most exact inversion formulas and methods discussed above are based (explicitly 
or implicitly) on some sort of the Green’s formula, Helmholtz representation, or 
eigenfunction decomposition for closed surfaces, and thus they cannot be extended 
to the case of partial data. The methods that do work in this situation rely on 
approximation techniques, as discussed below. 


Constant Speed of Sound 

Even the case of an acoustically homogeneous medium is quite challenging when 
reconstruction needs to be done from partial data (i.e., when the acquisition surface 
S is not closed). As it was discussed in section “Incomplete Data,” if the detectors 
located around the object in such a way that the “visibility” condition is not satisfied, 
accurate reconstruction is impossible: the “invisible” interfaces will be smoothed 
out in the reconstructed image. On the other hand, if the visibility condition is 
satisfied, the reconstruction is only mildly unstable (similarly to the inversion of the 
classic Radon transform) [71, 86]. If, in addition, the uniqueness of reconstruction 
from partial data is guaranteed (which is usually the case, see section “Uniqueness 
of Reconstruction”), one can hope to be able to develop an algorithm that would 
reconstruct quality images. 

Special cases of open acquisition surfaces are a plane or an infinite cylinder, for 
which exact inversion formulas are known (see, e.g., [16,34,41,96] for the plane and 
[101] for a cylinder). Of course, the plane or a cylinder would have to be truncated 
in any practical measurements. The resulting acquisition geometry will not satisfy 
the visibility condition, and material interfaces whose normals do not intersect the 
acquisition surface will be blurred. 

Iterative algebraic techniques (see the corresponding paragraph in section “Con- 
stant Speed of Sound”) were among the first methods successfully used for 
reconstruction from surfaces only partially surrounding the object (e.g., [14, 15,76]). 
As it is mentioned in section “Constant Speed of Sound,” such methods are very 
slow. For example, reconstructions in [14] required the use of a cluster of computers 
and took 100 iterations to converge. 

Parametrix-type reconstructions in the partial data case were proposed in [17]. A 
couple of different parametrix-type algorithms were proposed in [72,74]. They are 
based on applying one of the exact inversion formulas for full circular acquisition to 
the available partial data, with zero-filled missing data, and some correction factors. 
Namely, since the missing data is replaced by zeros, each line passing through 
a node of the reconstruction grid will be tangent either to one or to two circles 
of integration. Therefore, some directions during the backprojection step will be 
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represented twice and some only once. This, in turn, will cause some interfaces to 
appear twice stronger than they should be. The use of weight factors was proposed 
in [72, 74] in order to partially compensate for this distortion. In particular, in [72] 
smooth weight factors (depending on a reconstruction point) are assigned to each 
detector in such a way that the total weight for each direction is exactly one. This 
method is not exact; the error is described by a certain smoothing operator. However, 
the singularities (or jumps) in the image will be reconstructed correctly. As shown 
by numerical examples in [72], such a correction visually significantly improves 
the reconstruction. Moreover, iterative refinement is proposed in [72,74] to further 
improve the image, and it is shown to work well in numerical experiments. 

Returning to non-iterative techniques, one should mention an interesting attempt 
made in [78, 79] to generate the missing data using the moment range conditions 
for M (see section “Range Conditions”). The resulting algorithm, however, does 
not seem to recover the values well, although, as expected, it reconstructs all visible 
singularities. 

An accurate 2D non-iterative algorithm for reconstruction from data measured 
on an open curve S' was proposed in [59]. It is based on precomputing approxima- 
tions of plane waves in the region of interest Q by the single-layer potentials of the 
form 


/ ZAly —xDe410), 


S 


where p(y) is the density of the potential, which needs to be chosen appropriately, 
dl(y) is the standard arc length, and Z(t) is either the Bessel function Jo(t) or the 
Neumann function Y(t). Namely, for a fixed € one finds numerically the densities 
pe, (y) and psy (y) of the potentials 


Wikpep) = i. Jo(Aly — x|)pe.2(y)dl(y), (45) 


Wr, psy) = [ Yo(Aly —x|)pex (aly), (46) 


where A = |&|, such that 
W7 (x, pe.) + Wy (x, psy) & exp(—i€-x) forall x € Q. (47) 


Obtaining such approximations is not trivial. One can show that exact equality in 
(47) cannot be achieved, due to different behavior at infinity of the plane wave and 
the approximating single-layer potentials. However, as shown by numerical exam- 
ples in [59], if each point in Q is “visible” from S, very accurate approximations 
can be obtained, while keeping the densities p¢,; and p¢,y under certain control. 
Once the densities pg; and pg y have been found for all €, function f(x) can be 


easily reconstructed. Indeed, for the Fourier transform f (&) of f(x) 
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FO) = 5 [ foexp-ig-v as. 
one obtains, using (47) 
fers [ feo [wacx ) + Wy (x, pey)) dx 
ie . JAX, Pe, YY, Pey 
ae a | f FMC zee | perdi) 
2x Is Lo oly PET (Y y 
1 
+f | [ reomaly-sdax] ororaion, 8) 
TJS LJQ 


where the inner integrals are computed from the data g 


/ FOAL —x)ax = / so nUnar (49) 
Q RT 


i f@)YAly —x)dx = i sooriOndn (50) 
Q RT 


Formula (48), in combination with (49) and (50), yields values of f (€) for 
arbitrary &. Now f(x) can be recovered by numerically inverting the Fourier 
transform or by a reduction to an FBP inversion [63] of the regular Radon transform. 

The most computationally expensive part of the algorithm, which is computing 
the densities pg; and p¢,y, needs to be done only once for a given acquisition sur- 
face. Thus, for a scanner with a fixed S, the resulting densities can be precomputed 
once and for all. The actual reconstruction part then becomes extremely fast. 

Examples of reconstructions from incomplete data using this technique of [59] 
are shown in Fig. 11. The images were reconstructed within the unit square [—1, 1] x 
[—1, 1], while the detectors were placed on the part of the concentric circle of radius 
1.3 lying to the left of line x; = 1. We used the same phantom as in Fig. 10a; the 
reconstruction from the data with added 15 % noise is shown in Fig. 11b; Fig. 11c 
demonstrates the results of applying additional smoothing filter to reduce the effects 
of noise in the data. 


Variable Speed of Sound 

The problem of numerical reconstruction in TAT from the data measured on open 
surfaces in the presence of a known variable speed of sound currently remains 
largely open. One of the difficulties was discussed in section “Incomplete Data”: 
even if the speed of sound c(x) is non-trapping, it can happen that some of the 
characteristics escape from the region of interest to infinity without intersecting 
the open measuring surface. Then stable reconstruction of the corresponding 
interfaces will become impossible. It should be possible, however, to develop stable 
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Fig. 11 Examples of reconstruction from incomplete data using the technique of [59]. Detectors 
are located on the part of circular arc of radius 1.3 lying left of the line x; = 1. (a) Reconstruction 
from accurate data, (b) reconstruction from the data with added 15 % noise, (c) reconstruction from 
noisy data with additional smoothing filter 


reconstruction algorithms in the case when the whole object of interest is located in 
the visible zone. 

The generalization of the method of [59] to the case of variable speed of sound is 
so far problematic, since this algorithm is based on the knowledge of the open space 
Green’s function for the Helmholtz equation. In the case of a nonconstant c(x), this 
Green’s function is position dependent, and its numerical computation is likely to 
be prohibitively time-consuming. 

A promising approach to this problem, currently under development, is to use 
time reversal with the missing data replaced by zeros or maybe by a more clever 
extension (e.g., using the range conditions, as in [78, 79]). This would produce an 
initial approximation to f(x), which one can try to refine by fixed-point iterations; 
however, the pertinent questions concerning such an algorithm remain open. 

An interesting technique of using a reverberant cavity enclosing the target to 
compensate for the missing data is described in [28]. 


5 Final Remarks and Open Problems 


We list here some unresolved issues of mathematics of TAT/PAT, as well as some 
developments that were not addressed in the main text. 


1. The issue of uniqueness acquisition sets S (i.e., such that transducers distributed 
along S provide sufficient information for TAT reconstruction) can be considered 
to be resolved, for most practical purposes. However, there remain significant 
unresolved theoretical questions. One of them consists of proving an analogue of 
Theorem 5 for non-compactly supported functions with a sufficiently fast (e.g., 
super-exponential) decay at infinity. The original (and the only known) proof 
of this theorem uses microlocal techniques [7, 85] that significantly rely upon 
the compactness of support. However, one hopes that the condition of a fast 
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decay should suffice for this result. In particular, there is no proven analogue 
of Theorem 3 for non-closed sets S (unless S is an open part of a closed analytic 
surface). 

Techniques developed in [36] (see also [8] for their further use in TAT) might 
provide the right approach. 

This also relates to still unresolved situation in dimensions 3 and higher. 
Namely, one would like to prove Conjecture 1. 

2. Concerning the inversion methods, one notices that closed-form formulas are 
known only for spherical, cylindrical, and planar acquisition surfaces. The 
question arises whether closed-form inversion formulas could be found for any 
other smooth closed surface? It is the belief of the authors that the answer to this 
question is negative. 

Another feature of the known closed-form formulas that was mentioned 
before is that they do not work correctly if the support of the sought function 
F(x) lies partially outside the acquisition surface. Time reversal and eigenfunc- 
tion expansion methods do not suffer from this deficiency. The question arises 
whether one could find closed-form formulas that reconstruct the function inside 
S correctly, in spite of it having part of its support outside. Again, the authors 
believe that the answer is negative. 

3. The complete range description of the forward operator WV in even dimensions 
is still not known. It is also not clear whether one can obtain complete range 
descriptions for nonspherical observation sets S or for a variable sound speed. 
The moment and orthogonality conditions do hold in the case of a constant speed 
and arbitrary closed surface, but they do not provide a complete description of 
the range. For acoustically inhomogeneous media, an analogue of orthogonality 
conditions exists, but it also does not describe the range completely. 

4. The problem of unique determination of the speed of sound from TAT data is 
largely open. 

5. As it was explained in the text, knowing full Cauchy data of the pressure p (i.e., 
its value and the value of its the normal derivative) on the observation surface 
S leads to unique determination and simple reconstruction of /. However, the 
normal derivative is not measured by transducers and thus needs to be either 
found mathematically or measured in a different experiment. Thus, feasibility of 
techniques [12,25] relying on full Cauchy data requires further mathematical and 
experimental study. 

6. In the standard X-ray CT, as well as in SPECT, the local tomography technique 
(33, 56] is often very useful. It allows one to emphasize in a stable way 
singularities (e.g., tissue interfaces) of the reconstruction, even in the case of 
incomplete data (in the latter case, the invisible parts will be lost). An analogue of 
local tomography can be easily implemented in TAT, for instance, by introducing 
an additional high-pass filter in the FBP-type formulas. 

7. The mathematical analysis of TAT presented in the text did not take into account 
the issue of modeling and compensating for the acoustic attenuation. This 
subject is addressed in [22, 52, 62, 80, 83], but probably cannot be considered 
completely resolved. 
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8. Quantitative PAT: This chapter, as well as most other papers devoted to 
TAT/PAT, is centered on finding the initial pressure f(x). This pressure, which is 
proportional to the initial energy deposition, is related to the optical parameters 
(attenuation and scattering coefficients) of the tissue. The nontrivial issue of 
recovering these parameters, after the initial pressure f(x) is found, is addressed 
in the recent works [18, 29, 30]. 

9. The TAT technique discussed in the chapter uses active interrogation of the 
medium. There is a discussion in the literature of a passive version of TAT, 
where no irradiation of the target is involved [77]. 
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Abstract 


In this chapter a general mathematical model of Optical Coherence Tomography 
(OCT) is presented on the basis of the electromagnetic theory. OCT produces 
high-resolution images of the inner structure of biological tissues. Images are 
obtained by measuring the time delay and the intensity of the backscattered 
light from the sample considering also the coherence properties of light. The 
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scattering problem is considered for a weakly scattering medium located far 
enough from the detector. The inverse problem is to reconstruct the susceptibility 
of the medium given the measurements for different positions of the mirror. 
Different approaches are addressed depending on the different assumptions made 
about the optical properties of the sample. This procedure is applied to a full field 
OCT system and an extension to standard (time and frequency domain) OCT is 
briefly presented. 


1 Introduction 


Optical Coherence Tomography (OCT) is a noninvasive imaging technique pro- 
ducing high-resolution images of biological tissues. OCT is based on Low (time) 
Coherence Interferometry and takes into account the coherence properties of light 
to image microstructures with resolution in the range of few micrometers. Standard 
OCT operates using broadband and continuous wave light in the visible and 
near-infrared spectrum. OCT images are obtained by measuring the time delay 
and the intensity of backscattered or back-reflected light from the sample under 
investigation. Since it was first established in 1991 by Huang et al. [24], the 
clinical applications of OCT have been greatly improved. Ophthalmology remains 
the dominant one, initially applied in 1993 [17,41]. The main reason is that OCT 
has limited penetration depth in biological tissues, but high resolution. The theory 
of OCT has been analyzed in details in review papers [14, 16, 32, 36, 44] in book 
chapters [15, 19,42] and in books [4, 5, 10]. 

To derive a mathematical model for the OCT system, the scattering properties 
of the sample need to be described. There exist several different approaches to 
model the propagation of light within the sample: the radiative transfer equation 
with scattering and absorption coefficients [9, 38,45], Lambert—Beer’s law with the 
attenuation coefficient [39,46], the equations of geometric optics with the refractive 
index [7], and Maxwell’s equations with the susceptibility (or the refractive index) 
as optical parameters of the medium [6, 12,27, 37,43]. Also statistical approaches 
using Monte Carlo simulations are used [2, 11, 26,31, 40]. 

This chapter describes the propagation of the electromagnetic wave through the 
sample using Maxwell’s equations and adopts the analysis based on the theory of 
electromagnetic fields scattered by inhomogeneous media [8, 20]. The sample is 
hereby considered as a linear dielectric medium (potentially inhomogeneous and 
anisotropic). Moreover, the medium is considered weakly scattering so that the 
first-order Born approximation can be used and, as it is usually assumed in OCT, 
the backscattered light is detected far enough from the sample so that the far field 
approximation is valid. Starting from this model, different reconstruction formulas 
for special cases regarding the inner structure of the sample are presented. 

This chapter is organized as follows. In Sect.2, the principles of OCT and 
different variants of OCT systems are presented. Section 3 describes the solution 
of Maxwell’s equations and an appropriate formula for the measurements of OCT 
is derived. Given the initial field and the optical properties (the susceptibility) of the 
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sample, the solution of the direct problem is obtained in Sect. 4. An iterative scheme 
is derived in the last section for the reconstruction of the unknown susceptibility, 
which is the inverse problem of OCT. 


2 Basic Principles of OCT 


OCT is used to gain information about the light scattering properties of an object by 
illuminating it with some short laser pulse and measuring the backscattered light. 

The name “Optical Coherence Tomography” is motivated by the way the 
scattering data are measured: To get more precise measurements, the backscattered 
light is not directly detected, but first superimposed with the original laser pulse 
and then the intensity of this interference pattern is measured (this means that one 
measures the “coherence” of these two light beams). 

Experimentally, this is done by separating the incoming light at a beam splitter 
into two identical beams which travel two different paths. One beam is simply 
reflected by a mirror and sent back to the beam splitter, while the other beam is 
directed to the sample. At the beam splitter, the beam reflected by the mirror and 
the backscattered light from the sample are recombined and sent to the detector 
[16, 25,44]. See Fig. 1 for an illustration of this procedure. 

There exist different variants of the OCT regarding the way the measurements 
are done: 


Time and frequency domain OCT: In time domain OCT, the position of the 
mirror is varied and for each position one measurement is performed. On 
the other hand, in frequency domain OCT, the reference mirror is fixed and 
the detector is replaced by a spectrometer. Both methods provide equivalent 
measurements which are connected by a Fourier transform. 

Standard and full field OCT: In standard OCT, the incoming light is focused 
through objective lenses to one spot in a certain depth in the sample and the 
backscattered light is measured in a point detector. This means that to obtain 
information of the whole sample, a transversal-lateral scan has to be performed 


Fig. 1 Schematic diagram of 


the light traveling in an OCT 

system. The laser beam ; <_ 
Source — 

emitted by the source is pee —? 

divided at the beam splitter 


into two light beams; one is 
reflected at a mirror, the other tl 


Sample 


one backscattered from the 
sample. The superposition of 


the two reflected beams is i 


then measured at the detector 
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(by moving the light beam over the frontal surface of the sample). In full 
field OCT, the entire frontal surface of the sample is illuminated at once and 
the single point detector is replaced by a two-dimensional detector array, for 
instance by a charge-coupled device (CCD) camera. 

Polarization-sensitive OCT: In classical OCT setups, the electromagnetic wave is 
simply treated as a scalar quantity. In polarization-sensitive OCT, however, the 
illuminating light beams are polarized and the detectors measure the intensity 
of the two polarization components of the interfered light. 


There are also further modifications such as Doppler OCT and quantum OCT, which 
are not addressed here. In this chapter, the focus is mainly on time domain full field 
OCT, but also the others are discussed. 


3 The Direct Scattering Problem 


To derive a mathematical model for an OCT system, one has to describe on one 
hand the propagation and the scattering of the laser beam in the presence of the 
sample and on the other hand the way how this scattered wave is measured at the 
detectors. For the first part, the interaction of the incoming light with the sample can 
be modeled with Maxwell’s macroscopic equations. 


Maxwell's Equations 


Maxwell’s equations in matter consist of the partial differential equations 


div, D(t, x) = 4p(t, x), teR, xeR’, (la) 
div, B(t, x) = 0, teR, xeR’, (1b) 
10B 
curl, E(t, x) = — 2 ap pe Rix eh, (1c) 
c 
4 10D 
curl, H(t,x) = “7 I(t, x) + arvana) teR, xeR’, (1d) 
c c 


relating the following physical quantities (at some time ¢ € IR and some location 
x € R?): 


Speed of light c R 
External charge density p(t, x) R 
External electric current density J(t, x) R3 
Electric field E(t, x) R3 
Electric displacement D(t, x) R3 
Magnetic induction Bit, x) R3 


Magnetic field H(t, x) R3 
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Maxwell’s equations do not yet completely describe the propagation of the light 
(even assuming that the charge density ¢ and the current density J are known, there 
are only 8 equations for the 12 unknowns F, D, B, and #7). 

Additionally to Maxwell’s equations, it is therefore necessary to specify the 
relations between the fields D and E as well as between B and H. 

Let 2 C IR? denote the domain where the sample is located. It is considered as 
a nonmagnetic, dielectric medium without external charges or currents, this means 
that for all ¢ € IR and all x € Q the electric and magnetic fields fulfil the relations 


D(t,x) = E(t,x) + - x(t, x)E(t — t, x)drt, (2a) 
0 

B(t,x) = H(t, x), (2b) 

p(t,x) = 0, (2c) 

J(t,x) =0, (2d) 


where the function y : R x R* — IR**? (for convenience, x is also defined 
for negative times by y(t,x) = 0 fort < 0, x € MR?) is called the (electric) 
susceptibility and is the quantity to be recovered. The time dependence of y hereby 
describes the fact that a change in the electric field EF cannot immediately cause a 
change in the electric displacement D. Since this delay is quite small, it is sometimes 
ignored and y(t, x) is then replaced by 6(t) x(x). Moreover, the medium is often 
considered to be isotropic, which means that y is a multiple of the identity matrix. 

The sample is situated in vacuum and the assumptions (2) are modified by setting 
for allt € Randall x ¢ R?\Q 


D(t,x) = E(t, x), (3a) 
B(t,x) = H(t, x), (3b) 
p(t,x) =0, (3c) 
J@;x) =0, (3d) 


This simply corresponds to extend the Eq. (2) to R. x IR} and to assume y(t, x) = 0 
forallt ¢e R, x e R?\Q. 

In this case of a nonmagnetic medium, Maxwell’s equations result into one 
equation for the electric field F'. To get rid of the convolution in (2a), it is practical to 
consider the Fourier transform with respect to time. In the following, the convention 


f(@,x) = [- f(t, xje@'dt, 


for the Fourier transform of a function f with respect to t is used. 
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Proposition 1. Let E, D, B, and H fulfil Maxwell’s equations (1). Moreover, let 
assumptions (2) and (3) be satisfied. Then the Fourier transform E of E fulfils the 
vector Helmholtz equation 


2 
curl,curl, E(@, x) — (Cl + %(w,x))E(w,x) =0, we R, xe R’. (4) 
c 


Proof. Applying the curl to (1c) and using (1d) with the assumptions B = H and 
J = 0, yields 


1 dcurl, B (t,x) 1 02D 
= x)= -— 
at c? at? 


curl, curl, E(t, x) = 


(t,x). (5) 


The Fourier transform of (2a) and (3a) and the Fourier convolution theorem (recall 
that y is set to zero outside &2) imply that 


D(w, x) =(1+ f(w, x) E(w, x), forall w €R, xe R’. 


Therefore, the Eq. (4) follows by taking the Fourier transform of (5). oO 


Initial Conditions 


The sample is illuminated with a laser beam described initially (before it interacts 
with the sample) by the electric field E® : R x IR? > R? which is (together with 
some magnetic field) a solution of Maxwell’s equations (1) with the assumptions (3) 
for all x € R*. Then, it follows from the proof of the Proposition 1, for x = 0, that 


2 
curl, curl, E (w, x) _ © Ee, x) =0, weER, xeER’. (6) 
c 


Moreover, it is assumed that E does not interact with the sample until the time 
t = 0, which means that supp E(t, -) 9 Q = @ for all t < 0. 

The electric field E : IR x IR* — R® generated by this incoming light beam in 
the presence of the sample is then a solution of Maxwell’s equations (1) with the 
assumptions (2) and the initial condition 


E(t,x) = E©(t,x) forall t <0, xe€R’. (7) 


Since Maxwell’s equations for F in Proposition | are reformulated as an equation 
for the Fourier transform £, it is helpful to rewrite the initial condition in terms of F. 


Proposition 2. Let E (together with some magnetic field H) fulfil Maxwell’s 
equations (1) with the assumptions (2) and (3) and with the initial condition (7). 
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Then the Fourier transform of E — E) fulfils that the function @ > E(a, x)- 
Eo (@, x), defined on IR, can be extended to a square integrable, EEG 
function on the upper half plane H = {w € © | Sm(w) > O} for every x € R?. 


Proof. From the initial condition (7) it follows that E(t,x) — E(t,x) = 0 for 
allt < 0. Thus, the result is a direct consequence form the Paley—Wiener theorem, 
which is based on the fact that in this case 


Co 
E(o, x) — E(w, x) =f (E — E(t, xe dt 
0 


is well defined for all w € H and complex differentiable with respect tow € H. O 


Remark that the electric field E is uniquely defined by (4) and Proposition 2. 


The Measurements 


The measurements are obtained by the combination of the backscattered field from 
the sample and the back-reflected field from the mirror. In practice, see Fig. |, the 
sample and the mirror are in different positions. However, without loss of generality, 
a placement of them around the origin is assumed in the proposed formulation, in 
order to avoid rotating the coordinate system. To do so, the simultaneously illumi- 
nation of the sample and the mirror is suppressed and two different illumination 
schemes are considered. The gain is to keep the same coordinate system but the 
reader should not be confused with illumination at different times. 

Thus, the electric field E, which is obtained by illuminating the sample with the 
initial field E (that is E solves (4) with the initial condition (7)), is combined 
with EF, which is the electric field obtained by replacing the sample by a mirror and 
illuminating with the same initial field E. 

The mirror is placed orthogonal to the unit vector e3; = (0, 0, 1) through the point 
re3. As in (7), it is assumed that supp £ ) (t, -) does not interact with the mirror for 
t < 0, so that 


E,(t,x) = E(t,x) forall ¢<0, x eR’, (8) 


Then the resulting electric field E, : IR x R* — R? is given as the solution of 
the same equations as E (Maxwell’s equations (1) together with the assumptions 
(2) and initial condition (8)) with the susceptibility y replaced by the susceptibility 
x, of the mirror at position 7. One sort of (ideal) mirror can be described via the 
susceptibility y,(¢,x) = 0 for x3 > r and y,(t,x) = Cd(t)1 for x3 < r with an 
(infinitely) large constant C > 0. 

The intensity J, of each component of the superposition of the electric fields E 
and E, averaged over all time is measured at some detector points. The detectors 
are positioned at all points on the plane 
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. 23 423 
Detector D Detector D 


LLLLLILLLIXLLLLLL LLL w=d0 LM 


v3 = R+26 


Mirror 


1,72 > ©1,72 


Sample Q 


Fig. 2. The two scattering problems involved in OCT: On the left hand side the scattering of the 
initial wave on the sample Q; on the right hand side the reference problem where the initial wave 
E©) is reflected by a perfect mirror at a tunable position r € (—00, R). The two resulting electric 
fields, FE and E,, are then combined and this superposition EF + EF, is measured at the detector 
surface D 


D={xeR |x, =d} 


parallel to the mirror at a distance d > 0 from the origin. The mirror and the sample 
are both located in the lower half plane of the detector surface with some minimal 
distance to D. Moreover, the highest possible position R € (6, d — 26) of the mirror 
shall be by some distance 6 > 0 closer to the detector than the sample, this means 
(see Fig. 2) 


supx3< R—6 and re (-—o,R). (9) 


xEQ 


To simplify the argument, let us additionally assume that the incoming electric 
field E does not influence the detector after the time t = 0, meaning that 


E®(t,x)=0 forall t>0,xeD. (10) 


At the detector array, the data are obtained by measuring 
Co 
Tj (x) = |E;(t,x) + E,;(t,x)|?dt, xeD, j € {1,2,3}. (11) 
0 


In standard OCT, the polarization is usually ignored. In this case, only the total 
intensity J, = ee [,,; needs to be measured, see Sect. 5 for the reconstruction 
formulas in the isotropic case. 
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In this measurement setup, it is easy to acquire besides the intensity J, also 
the intensity of the two waves F and E, separately by blocking one of the 
two waves F and E, at a time. Practically, it is sometimes not even neces- 
sary to measure them since the intensity of the reflected laser beam E, can 
be explicitly calculated from the knowledge of the initial beam E), and the 
intensity of E is usually negligible compared with the intensity J, (because of 
the assumption (10), the field F contains only backscattered light at the detector 
after the measurement starts). Therefore, one can consider instead of J, the 
function 


1 lo, e} loo} 
Mr) = 5(ts- [le enpar- f"ie,e.mPar) a2 


forr € (—oo, R), j € {1,2,3}, and x € D as the measurement data. 
Proposition 3. Let the initial conditions (7) and (8) and the additional assump- 


tion (10) be satisfied. Then, for all x € D, r € (—oo, R), and j € {1,2,3} the 
measurements M,, defined by (12), fulfil 


M,,;(x) = [- (Ej — E\)(t,x)(Er,j — EO), x)dt (13a) 


lo, 
~ / (Ej — EP (eo, x)(Erj — E(w, x)do, (13b) 
—oo 
Proof. Expanding the function J, ;, given by (11), gives 
[o,) 
LjG)= / (\Ej(t. x)? + |E,j (t,x)? + 2E;(t,x)E,,; (t,x) dt. 
0 
Thus, by the Definition (12) of M,, it follows that 
CO 
M(x) = f Ej (t,x) E,;(t, x)dt, 
which, using the assumption (10), can be rewritten in the form 
_ ©) 0) 
M,,;(x) = i (Ej — Ej )@,x)(E,j — Ej )(t, x)dt. 
0 


Then, since E and E, coincide with E® for t < 0, see (7) and (8), the integration 
can be extended to all times. This proves the formula (13a) for M,. The second 
formula follows from Plancherel’s theorem. oO 
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4 Solution of the Direct Problem 


In this section the solution of the direct problem, to determine the measurements 
M,, defined by (13a), from the susceptibility 7, is derived using Born and far field 
approximation for the electric field. 


Proposition 4. Let E be a solution of the Eqs.(4) and (7). Then, the Fourier 
transform E solves the Lippmann—Schwinger integral equation 


2 
E(w, x) = E%(@,x)+ (Sa + grad, div«) i G(w,x—y)x(w, y)E(@, y)dy, 
Cc R3 


(14) 
where G is the fundamental solution of the Helmholtz equation given by 


Proof. Equation (4) can be rewritten in the form 
A w A 
curl,curl, E(w, x) — = E(@, x) = $(@, x) 
c 
with the inhomogeneity 
wo , A 
o(@, x) = or Al, x) E(o, x). (15) 


Using that E© solves (6), the difference E — E) satisfies the inhomogeneous 
vector Helmholtz equation 


2 
A A (69) A A 
curl,curl,(E — E®)(w, x) — =e = E)(w, x) = ¢(@, x). (16) 
c 
The divergence of this equation, using that div,curl, (E =f ()) = 0, implies 
: A B(0) ome 
div,(E — E“’)(@,x) = — pi divx (@, x). (17) 
Applying the vector identity 


curl,curl,(Z — E) = grad, div, (E — E®) — A,(E — E®) 


in (16) and using (17) yields 
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— i x Ae e ' 
NY lene ore )\(w, x) + a =p \(w, x) = =e eae (O-*) — d(@,x). 


This is a Helmholtz equation for E — E© and the general solution which is (with 
respect to w) holomorphic in the upper half plane (equivalent to (7) by Proposition 2) 
is given by, see [8] 


2 2 
(E — E)(@, x) = -<| G(w,x—y) (G1 + prad div, o(@, y)dy 
R3 ; 


oe re: 

Sa (1 + grad div; } / G(w,x — y)d(@, y)dy. 
o c R3 

For the last equality, integration by parts and grad, G(@, x — y) = —grad,,G(w, x — 

y) were used. The Lippmann—Schwinger equation (14) follows from the last 

expression inserting the expression (15) for ¢. oO 


This integral equation uniquely defines the electric field E. The reader is referred 
to [1,8] for the isotropic case and to [33] for an anisotropic medium. 


Born and Far Field Approximation 


To solve the Lippmann—Schwinger equation (14), the medium is assumed to be 
weakly scattering, which means that ¥ is sufficiently small (implying that the dif- 
ference E — E becomes small compared to E)) so that the Born approximation 
E), defined by 


2 
EV (@,x) = E(w, x) + (Sa + grad divs 
c 


[,G@.x- Ko. BC. »dy. (18) 


is considered a good approximation for the electric field E, see [3]. To describe 
multiple scattering events, one considers higher order Born approximations. For 
different linearization techniques, the reader is referred to [1,23]. Moreover, since 
the detector in OCT is typically quite far away from the sample, one can simplify 
the expression (18) for the electric field at the detector array by replacing it with its 
asymptotic behavior for |x| — oo, that is replace the formula for E™) by its far field 
approximation (the far field approximation could also be applied to the solution £ 
of the Lippmann—Schwinger equation (14)). 


Proposition 5. Consider, for a given function @ : R> — R? with compact support 
and some parameter a € R, the function 
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ialx—y| 


g:R?>R, g(x) = i; o(y)dy. 


3 |x — y| 
Then, it follows, asymptotically for p — oo and uniformly in 3 € S?, that 


a 


iap , 
(a? + grad, div.) g(p) ~ ——~ / ox (0 x H(y))ei@? dy (19) 
R3 


p 


Proof. Consider the function 


Then 


rT ( i= 0 ( ia 1 sx er 
OxjOx, °° Ox; [Nae ee) 
la 1 ia 1 1AX ; XK 
a i cell SY - j 
(ip r) os fe a) [x| 


1a 1 XjxXe |; 
+ (-2—. +3 ) ! jee 
( |x|? [x4 J |x| 


Therefore, writing x in spherical coordinates: x = pd with p > 0, 0 € S?, for 
p — co uniformly in #, it can be seen that 


“er aeiaP 1 
v= - 0 %+0(—), 
ax : po as (;:) 


The approximation (locally uniformly in y € IR) 


o 1 1 
[pv — y| = fio 200.3) + sly? =p-(9,y) +0(2). 
p p p 


implies that (again uniformly in # € S*) 
geia(e—(9.9) 1 
(pu -—y)= ee +O (<) 4 


Now, considering the compact support of ¢ and using that x € IR? \ supp¢ 


Ox; OX; 


a 0 ar 
i ; = = d 
(grad, div, g) ; (x) d Dx) ds ea (x — y)bx(y)dy 
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= 5 eon 
7 RS fa OX j OXx ai 


Asymptotically for |x| — oo (again using the compact support of @) one obtains 


: : 3 gia(o—(0,y)) 
a°gj(p0) + (grad, divsg);(p9) =a? [YY (8x ~ 89) bu(0)dy. 
"k= 


The approximation (19) follows from the vector identity } x (3 x ¢) = (0,6) 0 — 
||? and || = 1. Oo 


The application of both the far field and the Born approximation, this means 
Proposition 5 for the expression (18) of E™), that is setting a = w/c and ¢ = 
2+ ZE in Proposition 5, imply th totic behavi 
GX in Proposition 5, imply the asymptotic behavior 


; ‘ 2eitP 
E% (a, pd) E%(, pd)—2 S 


[90x Go. NEW. ype ay. 
. (20) 


Arpc? 


The Forward Operator 

To obtain a forward model for the measurements described in Sect. 3, the (approx- 
imative) formula (20) is considered as a model for the solution of the scattering 
problem. To make this formula concrete, one has to plug in a function E 
describing the initial illumination (recall that E has to solve (6)). 


The specific illumination is a laser pulse propagating in the direction —e;, 
orthogonal to the detector surface D = {x € IR? | x3 = d}, this means 


EO(t,x) = ft+=)p, (21) 


which solves Maxwell’s equations (1) with the assumptions (3) for some fixed vector 
p € R3, with p3 = (p, e3) = 0, describing the polarization of the initial laser beam. 


Proposition 6. The function E©, defined by (21) with (p, e3) = 0, solves together 
with the magnetic field H, defined by 


H%G,x) = f+ 2)p xes, 
Maxwell’s equations (1) in the vacuum, that is with the additional assumptions (3). 


Proof. The four equations of (1) can be directly verified: 
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1 
div, E (t,x) = —f'(t + #4) (e3, p) = 0, 
Cc 


1 
div, H(t, x) = ae + 2) (e3, p x e3) = 0, 


1 10H© 
curl, EO, x) = — f’(t + Jes x p = --——(t, x), 
Cc . c ot 
1 1 10E© 
curl, H(t, x) = —f'(t + Ses x (p x e3) = —f'(t+ S)p = 7 (t,x 
c c c 


To guarantee that the initial field E® (and also the magnetic field H) does 
not interact with the sample or the mirror for tf < 0 and neither contributes to the 
measurement at the detectors for t > O as required by (8) and (10) the vertical 
distribution f : IR — R should satisfy (see Fig. 2) 


suppf c (4, 4). (22) 


In the case of an illumination E of the form (21), the electric field E, produced 
by an ideal mirror at the position r is given by 


E,(t,x) = 


a Rens if.x3 > 1, (23) 


if x3 <7. 


This just corresponds to the superposition of the initial wave with the (orthogonally) 
reflected wave, which travels additionally the distance 2“. The change in 
polarization of the reflected wave (from p to —p) comes from the fact that the 
tangential components of the electric field have to be continuous across the border 
of the mirror. 

The following proposition gives the form of the measurements M,., described in 
Sect. 3, on the detector surface D for the specific illumination (21). 


Proposition 7. Let E© be an initial illumination of the form (21) satisfying (22). 
Then, the equations for the measurements M,. from Proposition 3 are given by 


M,.;(x) = -»; | (Ej — E\)(t, x) f(t + dr, (24a) 


j BS. A A A PO) — 
- al (Ej — EP (eo, x) f (-a)el®# deo (24b) 
—oo 


forall 7 € {1,2,3}, r € (—oo, R), and x € D. 
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Proof. Since the electric field E, reflected on a mirror at vertical position r € 
(—oo, R) is according to (23) given by 


E,(t,x) =(f(t+ 8)— ft + 4=8))p forall t€R, xe D, 


the measurement functions M, (defined by (12) and computed with (13a)) are 
simplified, for the particular initial illumination £ © of the form (21), to (24a) for 
xeD. 

Since formula (24a) is just a convolution, the electric field E — E © can be 
rewritten, in terms of its Fourier transform, in the form 


7 le. ) lo e) a m . 
M,.)(x) = -2 | / (B; — Bo, ne“ f(t + 22) dood. 
—oo J—0o 


Interchanging the order of integration and applying the Fourier transform f of ff it 
follows Eq. (24b). oO 


In the limiting case of a delta impulse as initial wave, that is for f(€) = 6(& —&) 
with some constant &) € (4, 4) satisfying (22), the measurements provide directly 
the electric field. Indeed, it can be seen from (24a) that 


M,,;(x) = —p; (Ej — EY )(2* + &, x). 


By varying r € (—oo, R), the electric field E' can be obtained (to be more precise, 
its component in direction of the initial polarization) as a function of time at every 
detector position. 

The following assumptions are made: 


Assumption 5. The susceptibility y is sufficiently small so that the Born approx- 
imation E“) for the solution E. of the Lippmann—Schwinger equation (14) can be 
applied. 


Assumption 6. The detectors are sufficiently far away from the object so that one 
can use the far field asymptotics (20) for the measured field. 


Under these assumptions, one can approximate the electric field by the far field 
expression of the Born approximation E“ and plug in the expression in (20) to 
obtain the measurements M,;, j € {1,2, 3}. 

The above analysis, introducing appropriate operators, can then be formulated as 
an operator equation. The integral equation (20) can be formally written as 


(EY — E)(a, x) = (Kof)(o. x), 


for a given E where the operator Ky : 1 E“ — E is given by 
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wee? 
(Kov)(, p98) = - 
TEPC 


goes. 


/ dx (8 x (V(@, YEO, ye TP dy, 
R3 


It is emphasized that % : R x R3 > C33 and E — EO : Rx R3 > C3, that 
is Kov is a function from R. x R? into C%. Equivalently, considering the Eq. (13b), 
one has 


M(r, x) = (M,j(2)),_, = (M(E® - £))(7,x), 


where the operator M is defined by 


3 


(Mv)(r, x) = ([- vj (ow, x)(E,j - EY Vw, x)d) , xeED. 


j=l 


Here, Mv is a function from R x D to R3. Thus, combining the operators Ky and 
M, the forward operator F : ¥ > M, F = MKo models the direct problem. The 
inverse problem of OCT is then formulated as an operator equation 


Fh=M. (25) 


For the specific illumination (21), one has 


E%(W,x) = ([. f+ seat) p= f(ojet™ p. (26) 


Then, the operators Ky and M simplify to 


eit? ot 
Koy(0.09) = -F =F flw) | (0x 1.9) Jet ay 7) 
R3 


and, recalling that M3 = 0 since p3 = 0 (the polarization in the incident direction 
is Zero), 


r oo oe a 2 
(My)(r, x) = (-2 | Vj (0.x) f-ayet* ae) : (28) 


j=l 

The operator Ky is derived from the Born approximation taking into account the 
far field approximation for the solution of the Lippmann—Schwinger equation (14). 
But, one could also neglect Assumption 5 and Assumption 6 and use the operator iC 
corresponding to Eq. (14), that is, 
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2 
(Kv)(@, x) = (G1 + grad, div.) / G(w,x — y)v(a, y)E(@, y)dy, 
Cc R3 


and considering the nonlinear forward operator F = MK. 

The next section focuses on the solution of (25), considering the operators K 
and M, given by (27) and (28), respectively. The inversion of ¥ is performed in 
two steps, first M is inverted and then Ko. 


5 The Inverse Scattering Problem 


In optical coherence tomography, the susceptibility 7 of the sample is imaged from 
the measurements M,(x), r € (—oo, R), x € D. Ina first step, it is shown that the 
measurements allow us to reconstruct the scattered field on the detector D, that is 
inverting the operator (28). 

The vertical distribution f : IR — R should additionally satisfy (see Fig. 2) 


suppf c (4,24 %)c (4,4) forsome 6 > 0. (29) 
This guarantees that the initial field E©® (and also the magnetic field H)) not 
interact with the sample or the mirror for tf < O and neither contribute to the 
measurement at the detectors for t > 0 as required by (8) and (10). 

The condition that the length of the support of E© is at most 26 (the assumption 
that the support starts at R is only made to simplify the notation) is required for 
Proposition 8. It ensures that the formula (13a) for the measurement data M,(x), 
x € D, vanishes for values r > R so that the integral on the right hand side of (30) 
is only over the interval (—oo, R) where measurement data are obtained (recall that 
measurements are only performed for positions r < R of the mirror). 


Proposition 8. Let E be an initial illumination of the form (21) satisfying (29). 
Then, the measurements M,. from Proposition 7 imply for the electric field E: 


‘A ms es 2 R 2 (or—x 
(E)- EP 0.9 flo)p) === [Maj (aye@2 ar (30) 
—CoO 


forall j € {1,2,3},@ € R, andx €D. 


Proof. Remark that the formula (24a) can be extended to all r € R by setting 
M,.;(x) = 0 for r > R. Indeed, from (29) it follows that E(t, -) = E(t, -) for 
allt < a Since F is a solution of the linear wave equation with constant wave speed 
c on the half space given by x3 > R — 6, the difference between E and E) caused 
by the sample needs at least time aoRt8 to travel from the point at x3 = R—6 to 
the detector at x3 = d, so: 


E(t,x) = E©(t,x) forall x € R? with x; =d andct <28+d-—R. 
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This means that the integrand vanishes for t < 2d R In the case of t > brd—R 
it holds for r > R that 


ct +2r—d>26+d—R+2R-—d=R+26, 


so that f(t + ard) = 0 by the assumption (29) on the support of f Therefore, for 
r > R, always one of the factors in the integrand in (24a) is zero which implies that 
M,(x) = Oforr > Rand x € D. 

Thus, Eq. (24b) holds for all r € IR and applying the inverse Fourier transform 


with respect to r, using that i (-—@) = f (@) because f is real valued, yields 
2,” _jdor BO) paren 
: M,,j(x)e" < dr = —pj(E; — E; )(@,x)f(@e", 
—oo 


which can equivalently be written as (30). oO 


This means that one can calculate from the Fourier transform of the measure- 
ments r +>» M,(x) at some frequency w the Fourier transform of the electric field 
at w as long as the Fourier transform of the initial wave E does not vanish at @, 
that is for 7 (w) 4 0. Thus, under the Assumption 5 and Assumption 6, Eq. (30) 
can be solved for the electric field E.. Proposition 8 thus provides the inverse of the 
operator M defined by (28). Now, the inversion of the operator Ky given by (27) is 
performed considering the optical properties of the sample. 


Proposition 9. Let E (t,x) be given by the form (21) with p3 = 0 and the 
additional assumption (29). Then, for every w € R \ {0} with f(w) 4 0, the 
formula 


a 870 pC . ~j2(2r— = 
pj[0 x (8 x Xo, 20 +e3))p)], ~ p i M,.,; (pitje te Or POs) dy 


o| fF)? 
(31) 
holds for all j € {1,2}, 8 € S? := {n € S? | n3 > 0}, and p = < (asymptotically 


for x — Oand p —> ov). 
Here x denotes the Fourier transform of x with respect to time and space, that is 


[o,) 
X(@,k) -| / x(t, x)e i) el dxdt =) Rw, xje Mh dx, (32) 
—oo JR3 R3 


Proof. Because of (26), the Fourier transform of the electric field E — E © can be 
approximated, using (20) with E ~ E“ (by Assumption 5 and Assumption 6), by 


wo f (ae? 


E — E®)(@, pd) ~ 
( )(w, pt) ee 


[oO 9 x 0 x Ho.”P ay. 
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Then, applying (32), one obtains 


(£ — E(w, pd) ~ efoe x (8 x Zw, 2(8 +e3))p). 33) 


From (30), it is known that, for fo) # Oand p; # 0, 


-i¢f (2r—pd3) gy 


(E; — E(w, pd) = - 


rj 


picf (@) 


This identity together with (33), asymptotically for w 4 0, yields the statement 
(31). Oo 


To derive reconstruction formulas, Proposition 9 is used, which states that from 
the measurements M, (under the Assumption 5 and Assumption 6) the expression 


pj[0x (0x H(@, 20 +e3))p)],, f= 1,2, (34) 


can be calculated. Here, p € IR? x {0} denotes the polarization of the initial 
illumination E®, see (21), and 3 € st is the direction from the origin (where 
the sample is located) to a detector. 


The Isotropic Case 


This section analyzes the special case of an isotropic medium, meaning that the 
susceptibility matrix y is just a multiple of the unit matrix, so in the following x is 
identified with a scalar. 

Then, from the sum of the measurements M,.; and M,.2, using the formula (31), 
one obtains the expression 


X(w, (9 + e3)) (p, 0 x (x p)) = Fw, 2(9 + e3)) (9, p)? — |pl’). 


Since (i, p)* < |p|? for every combination of p € R? x {0} and 3 € S2, one has 
direct access to the spatial and temporal Fourier transform 


Xo, 2 +e3)), wo ER\ {0}, ve Si, (35) 


of x ina subset of R x R?. 

However, it remains the problem of reconstructing the four-dimensional suscep- 
tibility data y from the three-dimensional measurement data (35). In the following, 
some different additional assumptions are discussed to compensate the lack of 
dimension, see Table 1. 
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Table 1 Different assumptions about the susceptibility and the corresponding reconstruction 
formulas 


Assumptions Reconstruction method Section 
x(@,k) = x(k) Reconstruction from partial (three | Non-dispersive Medium in 
dimensional) Fourier data: Full Field OCT 
x(k), k ER’, 
X(k,e3) € (— 4. 4) 
x(@,k) = x(k) Reconstruction from full (one Non-dispersive Medium with 
dimensional) Fourier data: Focused IIlumination 
X(ks), k3 € R \ {0} 
supp x(-,x) C [0, 7] Recursive formula to get limited | Dispersive Medium 
R(x(t, -))(-, g) is angle Radon data 
piecewise constant R(x(t, -))(G,9),0 €R, gy € S? 
with £(g, e3) € (—4. 5) 
Xx) = Recursive formula to Dispersive Layered Medium 
5(x1)6(x2) x(t, x3), reconstruct 7 with Focused Illumination 


supp x(-,x) C [0, T], and 
X(t, +) is piecewise constant 
Here, (Rg)(o, 9) = Jecer3|(x.)=03 g(y)ds(y), o € R, gy € S’, denotes the Radon transform 


of a function g: R? > R. 


Non-dispersive Medium in Full Field OCT 

The model is simplified by assuming an immediate reaction of the sample to the 
exterior electric field in (2a). This means that x can be considered as a delta 
distribution in time so that its temporal Fourier transform ¥ does not depend on 
frequency, that is ¥(w, x) = X(x). Thus, the reconstruction reduces to the problem 
of finding ¥ from its partial (spatial) Fourier data 


x(k) for ke{2{W+e)eR*|F eS, o€E R\ {0} 


= {x € R? \ {0} | arccos((r7,€3)) € (—F, Z)}- 


Thus, only the Fourier data of y in the right circular cone C with axis along e3 
and aperture + are observed (see Fig. 3). In practice, these data are usually only 
available for a small range of frequencies w. 

Inverse scattering for full field OCT, under the Born approximation, has been 
considered by Marks et al. [28,29] where algorithms to recover the scalar suscepti- 


bility were proposed. 


Non-dispersive Medium with Focused Illumination 

In standard OCT, the illumination is focused to a small region inside the object so 
that the function y can be assumed to be constant in the directions e; and e (locally 
the illumination is still assumed to be properly described by a plane wave). Then, 
the problem can be reduced by two dimensions assuming that the illumination is 
described by a delta distribution in these two directions. As before, 7 is assumed to 
be frequency independent, so that 


X(@, x) = 8(x1)6 (x2) ¥ (x3), 
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Q 


> ki, ke 


Fig. 3 RegionC = R Xx (Si. + e3) of the available Fourier data of 


this means that the spatial and temporal Fourier transform (32) fulfils 7(@,k) = 
X(k3). In this case, the two-dimensional detector array can be replaced by a single 
point detector located at de3. 

Then, the measurement data (35) in direction ? = e3 provide the Fourier 
transform 


x72) forall we€R \ {0}. 


Therefore, the reconstruction of the (one dimensional) susceptibility x3 > ¥(x3) 
can be simply obtained by an inverse Fourier transform. 

This one-dimensional analysis, has been used initially by Fercher et al. [18], 
reviewed in [13] and by Hellmuth [22] to describe time domain OCT. Ralston et al. 
[34,35] described the OCT system using a single backscattering model. The solution 
was given through numerical simulation using regularized least squares methods. 


Dispersive Medium 
However, in the case of a dispersive medium, that is frequency-dependent 7, the 
difficulty is to reconstruct the four-dimensional function x : IR x R3 > C from the 
three-dimensional data 


m:RxSi,>C, mMo@,0) = X(w, 2(8 + e3)). (36) 


Lemma 1. Let m be given by (36). Then its inverse Fourier transform m : R. x 
Ss rl — C with respect to the first variable is given by 


m(t, 3) = —t,8)dr, teR, de Si, (37) 


Cc Oe 
2 +95) [. ast 
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Detector D 


Evnsiyt,0 


Ent,s 


71,72 
E(n-1)T,9 


Fig. 4 Discretization of x with respect to the detection points 


where 


Hei0.8)= fxr y)ds(y), BOER VES. 


Eo.9 


and Ey denotes the plane 


Eos = {y € R? | (8 +e3,y) =co}, cE R ve S:, (38) 


Proof. Taking the inverse temporal Fourier transform of and using (32), it follows 
that 


oe) 
m(t, 3) = =| X(w, 2(8 + e3))e "dw 
TT J—oco 
1 ee ii : 
= =| / F(w, xe te P+) eit dy da, 
—oo JR3 


Interchanging the order of integration, the integral over w is again described by an 
inverse Fourier transform and the previous equation becomes 
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m(t, 0) = / x(t + i (0 + e3,x),x)dx. 
R3 


Substituting then the variable x3 by tT = tf + i (3 + e3, x), this can be written as 


m(t, 3) = </ i / X(t, Wr-1,9 (%1, X2))dx1dxodt (39) 


with the function 


0 v 
Wo.3 7 R? => R’, Wo. (X1, X2) = (sx co 1X1 + =) 


1403 1+ 0 


Now, Wo.» is seen to be the parametrization of the plane 


Cal 

T+05 be 
Eo» = {y € R? | (vg, y) = ao, ith vy =| d  dg.9 = ——., 
o=ty | (vx, ¥) =do.9} with vy re and 4o9 = 73, 


see Fig. 4. The square root of the Gram determinant of the parametrization Wo,» is 


now given by the length of the vector vy = “tos x aot which implies that 
[eve) lee) 1+ 03 
[fe toGrn)ande =~" fre». 
—CO ¥—00 2 E,-1.9 
Plugging this into (39) yields the claim. oO 


Thus, the measurements give the combination (37) of values y of the Radon 
transform of the function y(t, -). It seems, however, impossible to recover the 
values ¥(t; 0, 7) from this combination 


m(t, 0) = X(t; t —t, B)dt, 


Cc oo 
¥ 2(1 + D3) [. 


since (for every fixed angle ? € 57) one would have to reconstruct a function on 
R? from one dimensional data. 

To overcome this problem, the function 7(t; -, #) is going to be discretized for 
every t € Rand # € S*, where the step size will depend on the size of the support 
of x(-, x). 

Let us therefore consider the following assumption. 


Assumption 7. The support of x in the time variable is contained in a small 
interval [0, T] for some T > 0: 
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suppx(-.x) C [0,7] forall x €R?. 


Then, the following discretization 
An(t, 0) = i y(t, y)ds(y), neZ, 1 €(0,T), dE S2, 
Ent 


of the Radon transform of the functions x(t, -) is considered, where E,,5 denotes 
the plane defined in (38). 


Assumption 8. The value ¥,(t, 0%) is a good approximation for the integral of the 


function x(t, -) over the planes Enr+¢,9 for all ¢ € [-4, ) (see Fig. 4), that is 


Xn(t, 9) & / x(t, y)ds(y), e€ [-5,4),n€Z, r€(0,T), VES}. 
EnT+6,9 


Under the Assumption 8, Eq. (37) can be rewritten in the form 


T 
m(t, 0) & / XN(c-1) (tT, B)dr, 
0 


c 
V2(1 + 33) 


where N(o) = |< + 5 | denotes the integer closest to +. This (approximate) 
identity can now be iteratively solved for y. 


Proposition 10. Let 


T 
m(t, 0) / Xn(r-t) (Tt, v)dr, veE s, teR, 
0 


Cc 
~ 21 + 93) 


for some constant T > 0 with the integer-valued function N(o) = | + 5 |. 
Then, x fulfils the recursion relation 


Xn(t,P) = Xngi(t, 9) + «-@+5)7,9), 


V2 + 83) om 
c ot 
neZ,té(0,T), dE Si. (40) 

Proof. Lett = —nT +e withe € [-4, ) andn € Z, then 


: T 
Ve=n= n fre (0,5 + €); 
n+1 ift e[£+.e,7). 


Taking an arbitrary ¢ € [-4, ) and n € Z, m can be formulated as 
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Tie T 
Cc 2 
n(—nT + 0) = + __| | Xn ,B)d +f Xn ,v)d : 
ae a som (/ 4 ) ‘ 


Differentiating this equation with respect to «, it follows that 


(Xn(4 + &, v) = Xn4iG + &, D)) , 


am 
AT = 
ar nT + 6,0) 


é 
J2(1 + 3) 
which (with tT = f + €) is equivalent to (40). oO 


Thus, given that 7,(¢,3) = O for sufficiently large n € Z (recall that supp 
x(t, -) C Q for all t € (0, T)), one can recursively reconstruct 7, to obtain the data 


/ x(t, y)ds(y) forall 1+ €[0,T), cE R, Be Sp 
Eo 


for the Radon transform of y(t, -). 

However, since the plane F,.9 is by its Definition (38) orthogonal to the vector 
0 +e, ford ¢€ Si, this provides only the values of the Radon transform 
corresponding to planes which are orthogonal to a vector in the cone C, see Fig. 3. 
For the reconstruction, one therefore still has to invert a limited angle Radon 
transform. 


Dispersive Layered Medium with Focused Illumination 
Except from ophthalmology, OCT is also widely used for investigation of skin 
deceases, such as cancer. From the mathematical point of view, this simplifies the 
main model since the human skin can be described as a multilayer structure with 
different optical properties and varying thicknesses in each layer. 

Here the incident field is considered to propagate with normal incidence to the 
interface x3 = L and the detector array is replaced by a single point detector located 
at de3. The susceptibility is simplified as 


X(t, x) = 8(x1)86(x2) x4, x3), 


and therefore the measurements provide the data, see (37) with ¥(w,k) = ¥(@, ks), 


m(o) = x(@, £2e3), w € R\ {0}. 


Considering the special structure of a layered medium, the susceptibility is 
described by a piecewise constant function in x3. This means explicitly that y has 
the form 


X(t, x3) = ta 0, ta 0.2] n=1,...,N (41) 


Xn(t), X3€ [Ln. Ln41) , 
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with (unknown) parameters L = L; > Lo >... > Ly+1 = O characterizing the 
thicknesses of the N layers and (unknown) functions 7,,. 
Lemma 1, for ? = e3, gives 


Cc 
m(t) = al X(t;t —t)dt, where y(t;0) = x(t, S). 
—oo 


Remarking that y is piecewise constant (41) and additionally assuming that 
x(-, x3) has compact support, see Assumption 7, with T < 2 min, (Ln — Ln+t) 
Proposition 10 can be applied for 3} = e3 to iteratively reconstruct y starting from 
Xo = 0. 


Modified Born Approximation 

In the proposed iteration scheme, Proposition 10, the traveling of the incident field 
through the sample before reaching a “specific” layer, where the susceptibility is 
to be reconstructed, is not considered. To do so, a modified iteration method is 
presented describing the traveling of the light through the different layers using 
Frensel’s equations. 

The main idea is to consider, for example, in the second step of the recursive 
formula, given 7, to find 72, as incident the field BO, given by (26), traveled also 
through the first layer. This process can be continued to the next steps. 

Let us first introduce some notations which will be used in the following. The 
fields E® and E® denote the reflected and the transmitted fields, with respect to 
the boundary L,,, respectively. The transmitted field EO after traveling through the 
n-th layer is incident on the L,,4 ; boundary and is denoted by aa The reflected 


field by the L,,+41 boundary back to the L,, boundary will be denoted by Foie and 


by Be) after traveling through the n-th layer (see Fig. 5). To simplify this model, 
multiple reflections are not included and the electric fields are taken to be tangential 
to the interface planes. 


Lemma 2. Let the sample have susceptibility given by (41) and let p, and t, denote 
the reflection and the corresponding transmission coefficients for the L,, boundary, 
respectively. Then, the field incident on the n-th layer with respect to the initial 
incident field EO := Ee is given by 


A (0) E© 
& => (M, ~M>: ao Moy (F:9) for n=N- | Pees = 
1 


assuming no backward field in the n-th layer, where 


My = — 


Tn 


1 ( aw An +1 (Ln-Ly+1) | 


12 tet Maka) gi Vt FUL Ln $1) 
Pne © e¢ 
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Layer 2, x2 


Ls ft}. In-1 
: Bt) 


Layer n — 1 


Ln A) 
a L 


n 


Layer n, Xn 


Layer n 
DEn+41 = 
: Ln+1 
In-1 . 
Layer N —1,xn-1 L : 
N-1 
In Layer N — 1 
© : 
Layer N, x Layer N "” 
Ly4i=O0 ——— ids 


1,22 


Fig. 5 Layered medium. Propagation of the initial field through the sample incident on the second 
layer (/eft image) and in general on the n-th layer, forn = 3,..., N (right image) 


Proof. Because of the assumptions (normal incidence, E® tangential to the 
boundary) the boundary conditions require the continuity of the total (upward 
and downward) electric and magnetic fields. Then, the reflection p, and the 
corresponding transmission t, coefficients for the L, boundary in terms of the 
susceptibility are given by [21] 


_ Vxn-1 $1 -— Sn +1 
Vin + 1+ S¥n + 1° 


Pn Th = 1+ Pn- 


To determine the propagation equations for the electric fields, the transfer matrices 
formulation is applied [30]. In particular, the fields at the top of the n-th layer can 
be computed with respect to the fields at the top of the (n + 1)th using 


a) _ 4, (88 
a(r) | = Mal acy for n=N-—1,...,1. 
En Eni 


and with respect to the incident field, 
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From the previous result, given 7, (by the recursion relation of Proposition 10), 
the matrix M,,+41 is computed to obtain the update ee which is then incident to 


the rest part of the sample. This means that E) is replaced by rea in the derivation 
of the measurements and the recursion relation (Lemma | and Proposition 10) for 
computing 7,41. For example, in the second step to reconstruct 72, the incident 
field is simply given by 


EO = ete VOTE Le) FO, 


The only unknown in this representation is the boundary L2 which can be 
approximated considering the point where change in the value of the measured 
function m is observed. The following analysis can be also extended for anisotropic 
media, but in a more complicated context since the displacement D and the electric 
field E are not always parallel. 

A simplification usually made here is to consider the sample field as the sum 
of all the discrete reflections and neglect dispersion. This mathematical model was 
adopted by Bruno and Chaubell [7] for solving the inverse scattering problem of 
determining the refractive index and the width of each layer from the output data. 
The solution was obtained using the Gauss—Newton method and the effect of the 
initial guesses was also considered. 

In conclusion, the traveling of the scattered field from the n-th layer through the 
sample could also be considered. Since the spherical waves can be represented as 
a superposition of plane waves by using similar techniques, in a more complicated 
form, one can obtain the transmitted scattered field. 


The Anisotropic Case 


In the anisotropic case, the susceptibility 7 cannot be considered a multiple of the 
identity. Therefore, the problem is to reconstruct from the expressions 


pj[d x (8 x xo, 29 +3))p)],,, j = 1,2, 


see (34), the matrix-valued function y : R x IR? — IR3*3, where it is assumed 
that measurements for every polarization p € IR? x {0} of the initial field E© are 
available. 

Introducing in analogy to (36) the function 


Mp Rx SLC, mp j(0,0) = Xopj(@, 2(8 + 63)), 


where Z»,p,; is for every } € S%, p € R* x {0}, and j € {1,2} the (spatial and 
temporal) Fourier transform of 


Nop; RxXRISR, yop;(t,x) = pj[d x (0 x x(t.x)p)],. 
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Lemma | (with m replaced by m, ; and x replaced by 7»,,,;) can be applied to find 
that the inverse Fourier transform of m,,; with respect to its first variable fulfils 


ee yee / . [ ans (Yds) 


Now, the same assumptions as in the isotropic case are considered, namely 
Assumption 7 and similar to Assumption 8: 


Assumption 9. The approximation 


; a GIT © i; topi% yds) forall e€(-Z,t 
Enr.o EnT+6,0 


is foreverytER ve Sai néZ, p € R? x {03, and j € {1,2} justified. 


Then, Proposition 10 provides an approximate reconstruction formula for the 
functions 


Zpqttso,8) = i Kops (t.y)ds() = pj [8 x (8x R(t10,8)p)], (42) 


Eo. 


forall p<€ R?x{0},reRoeRve S7,and j € {1,2}, where 


i(c:0,9) = [ x(x, y)ds(y) (43) 


denotes the two-dimensional Radon transform data of the function y(t, - ). 


Proposition 11. Let 3 € Ss? be fixed and ap,j;, p € R? x {0}, 7 = 1,2, be such 
that the equations 


pj[v x (8 x Xp)]j = ay; forall p« R’x {0}, 7 € {1,2}, (44) 


for the matrix X € IR**? have a solution. 
Then X € IR3*? is a solution of (44) if and only if 


(PX) = Bu. B= ( de de A oo) | kobe {1,2}, 
Qes,2 — Qe, +e2,2 —Ae, 2 
(45) 


where Ps € IR°*3 denotes the orthogonal projection in direction 9. 


Proof. First, remark that the equation system (44) is equivalent to the four equations 
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de, = [0 x (8 x Xehi, Ae, +er1 = Ae,1 + [0 x (0 x Xer)|1, 


(46) 
de .2 = [v x (0 x Xe2)]2, Ge, +e7,.2 = Aen2 + [0 x (0 x Xe,))2, 


which correspond to the Eq. (44) for (p, 7) € {(e1, 1), (€2, 2), (e1 + e2, 1), (e1 + 
2, 2)}. Indeed, for arbitrary polarization p = p,e; + p2é2, the expression p; [0 x 
(3 x Xp)]; can be written as a linear combination of the four expressions [0 x (0 x 
Xe;)|x, i,k = 1,2: 


pild x (8 x Xp) = pid x (8 x Xevhi + pipald x (8 x Xer)hi, 
pr[d x (8 x Xp)l2 = pipald x (0 Xero + ppl x (9 x Xer)h. 
and is thus determined by (46). 
Now, the equation system (46) written in matrix form reads 


(9 x x xo =-[2 (21) | , Keno, (47) 
P2) Jy 


for all p € R? x {0} with B defined by (45). 
Decomposing Xp = (03,Xp)0 + P Xp, where Pp € IR3*? denotes the 
orthogonal projection in direction 7, and using that 


vx (0x Xp) =0 x (0 x Po Xp) = (0, Pp» Xp) 0 — Py Xp = —Po9 Xp, 
the Eq. (47) can be written in the form (45). oO 


Proposition 11 applied to the Eq. (42) for the matrix X¥ = y(t;0, 0) for some 
fixed values t,0 € IR and } € Sj shows that the data ap; = pj[¥ x (0 x 
X(t; 0,0))]; for 7 = 1,2 and the three different polarization vectors p = e1, 
P = 2, and p = e; + e2 uniquely determine with Eq. (45) the projection 


7 ie [ (Pox. y)eeds(y) for k,£€ {1,2}. 


Moreover, measurements for additional polarizations p do not provide any further 
information so that at every detector point, corresponding to a direction ? € S ca 
only the four elements (P37)x2, k,£ = 1,2, of the projection Py» x influence the 
measurements. 

To obtain additional data which make a full reconstruction of y possible, one can 
carry out extra measurements after slight rotations of the sample. 

So, let R € SO(3) describe the rotation of the sample. Then the transformed 
susceptibility 7p is given by 


xr(t,y) = Ry(t, R'y)R’. (48) 
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Lemma 3. Let x: Rx R* > R**? be the susceptibility of the sample and 3 € s2 
be given. Furthermore, let R € SO(3) be such that there exists a constant dr > 0 
and a direction Bp € s2 with 


Dr +e3 =arR(0 4+ e3) (49) 


and define the susceptibility xp of the rotated sample by (48). 
Then, the data 


XR.p.j (750, VR) = pj Pr x (+. xf gates) , (50) 


OUR J 


corresponding to the measurements of the rotated sample at the detector in direction 


Tp, see (42), fulfil that 


XRpj (tT: CRO, OR) = Pj [Or X (Or x RY(t30, 9)R']; (51) 


forall t,o € R, p € R? x {0}, j = 1,2, where x is given by (43). 


Proof. Inserting Definition (48) and substituting z = Ry, formula (50) becomes 


Xrp.j (Ti 0, BR) = pj x (*. xf Rx, oat) . 
RT Esa p . 


J 


Since now, by the Definition (38) of the plane E,.y and by the Definition (49) of %, 


R'E cvp _ {RTy € R? | (UR + é3, y) = co} 
(R' (8p + e3),z) = co} 


w 


={zeEeR 


= {ze R3 QR (0 + e3, 2) =co}= Exo, 


it follows (51). oO 


This means that the data ¥pr,p,;(t; aro, 02) obtained from a detector placed 
in the direction tr, defined by (49), depends only on the Radon transform data 
X(t; o, 3). However, it still remains the algebraic problem of solving the Eq. (51) 
for different rotations R to obtain the matrix 7(t; 0, 0) € R33, 


Proposition 12. Let A ¢ R**? and vd € ea be given. Moreover, let Ro, R\, Ro € 
SO(3) be rotations so that every proper subset of {Rje3, Ries, R3e3, 0 + e3} is 
linearly independent and such that there exist for every R € {Ro, Ri, Ro} constants 
ap > Oand Jp € S% fulfilling (49). 


1200 P. Elbau et al. 


Let further P_ € IR?*? be the orthogonal projection in direction e3, Pg € R*? 


the orthogonal projection in direction 0 € IR*, and 


B= ( Re 1 QR e,.1 — eas) 
QR e7,2 — AR e,+e9,2 aR 2 


QR, pj => Pj [DR x (OR x RAR'p)];, 


for every R € {Ro, Ry, Ro}. 
Then, the equations 


PP»5,RXR'P' = Br, RE{Ro, Ri, Ro} (52) 
have the unique solution X = A. 
Proof. Using that dr = arR(0 + e3) — e3, see (49), it follows with Pe; = 0 that 
PP», = P(1— 001) = P —argPR(O + €3)01. 
With this identity, the Eq. (52) can be written in the form 


PR(X —ar(0 + €3)02RX)R'P! = Br. (53) 


Let now nr € RR? denote a unit vector orthogonal to 3 + e3 and orthogonal to 
Re. Then Rye is orthogonal to e3 and therefore, with P'P= Peas 


(PRyr)' PR = (Pe, Rnr)' R = (Ryr)'R = np. 
Thus, multiplying the Eq. (53) from the left with (PR r)", it follows that 
Nr(X — dr(0 + €3)0~RX)R'P* = (PRnp)' Br. 
Since now naz is orthogonal to % + e3, this simplifies to 
NpXR'P* = (PRnpr)' Br. (54) 
Remarking that the orthogonal projection onto the line IRR can be written as the 


composition of two orthogonal projections onto orthogonal planes with intersection 
Rn R: 


MRR = Py, Pores; 


where vp is a unit vector orthogonal to nr and 3 + e3, and multiplying Eq. (54) 
from the left with nr, one finds that 
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T pT T 
Py (Pete; X)R Po= NR(PRyr) Br. 


Applying then the projection P from the right and using that R'P™P = R'™P,, = 
Prt, R’, this can be written in the form 


Py_(Po+e,X) Pere, = Nr(PRnR)' BrPR. (55) 


Evaluating this equation now for R = Ro, Ri, Ro and remarking that {vpa,, vR,, VR} 
and {Rje3, Rj e3, R5e3} are linearly independent, one concludes that the 3x3 matrix 
P5+4e,X is uniquely determined by (55). 

However, it is also possible to calculate X (and not only its projection P+, X) 
from Eq. (53). Because 


X = Po4e,X + ————- (0 0 e 
it follows from (53) that 
0+ e3 
PR | ———~(1 — ard} RIO 0 Tx )R'P! 
(Gas ARVRR(O + e3))(U + e3) ) 


= Br— PR(L—ar(0 + €3)0RR)Pi4e,XR'P*. (56) 
Plugging in the identity 
aRORR(D + €3) = Ban + es) = 1+ ORs, 
which follows from the Definition (49) of 3}, applying P' from the left and P from 


the right, and using as before R' P™P = Par,,R", the Eq. (56) yields 


0 +e 


— Ur3P rte, (i 


(0 + ex) Prte, 
= R'P'BrPR— Parte,(1L—ar(0 + €3)0R) Po4e;XPpte,- (57) 


Since the right hand side is already known (it depends only on P»+¢,X), the 
equation system (57) for R = Ro, R,, Rz can be uniquely solved for 


v+e 
ace” + e3)'X. 


Therefore, the Eq. (52) uniquely determine X and because A is by construction 
a solution of the equations, this implies that XY = A. Oo 
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Thus, applying Proposition 12 to the matrix A = x(t; 0,3) shows that the mea- 
surements dp »,; obtained at the detectors 0p for the polarizations p = e1, €2,e; + 
é, and rotations R = Ro, Rj, Ro, fulfilling the assumptions of Proposition 12 
provide sufficient information to reconstruct the Radon data x(t; 0, 3). Calculating 
these two-dimensional Radon data for all directions } in some subset of ST (by 
considering some additional rotations so that for every direction #, there exist three 
rotations fulfilling the assumptions of Proposition 12), it is possible via an inversion 
of a limited angle Radon transform to finally recover the susceptibility y. 


6 Conclusion 


In this chapter, a general mathematical model of OCT based on Maxwell’s equations 
has been presented. As a consequence of this modeling, OCT was formulated as 
an inverse scattering problem for the susceptibility v. It was shown that without 
additional assumptions about the optical properties of the medium, in general, y 
cannot be reconstructed due to lack of measurements. Some reasonable physical 
assumptions were presented, under which the medium can, in fact, be reconstructed. 
For instance, if the medium is isotropic, iterative schemes to reconstruct the sus- 
ceptibility were developed. Dispersion and focus illumination are also considered. 
For an anisotropic medium, it follows that different incident fields, with respect to 
direction (rotating the sample) and polarization, should be considered to completely 
recover y. 


Acknowledgments The authors would like to thank Wolfgang Drexler and Boris Hermann from 
the Medical University Vienna for their valuable comments and stimulating discussions. This work 
has been supported by the Austrian Science Fund (FWF) within the national research network 
Photoacoustic Imaging in Biology and Medicine, projects $10501-N20 and $10505-N20. 


Cross-References 


Inverse Scattering 
Optical Imaging 
Tomography 
Wave Phenomena 


References 


1. Ammari, H., Bao, G.: Analysis of the scattering map of a linearized inverse medium problem 
for electromagnetic waves. Inverse Prob. 17, 219-234 (2001) 

2. Andersen, P.E., Thrane, L., Yura, H.T., Tycho, A., Jorgensen, T.M., Frosz, M.H.: Advanced 
modelling of optical coherence tomography systems. Phys. Med. Biol. 49, 1307-1327 (2004) 

3. Born, M., Wolf, E.: Principles of Optics. 7th Edn. Cambridge University Press, Cambridge 
(1999) 

4. Bouma, B.E., Tearney, G.J.: Handbook of Optical Coherence Tomography. Marcel Dekker 
Inc., New York (2002) 


Mathematical Methods of Optical Coherence Tomography 1203 


5. 


6. 


7. 


8. 


24. 


25. 


26. 


27. 


28. 


29. 


Brezinski, M.E.: Optical Coherence Tomography Principles and Applications. Academic 
Press, New York (2006) 

Brodsky, A., Thurber, S.R., Burgess, L.W.: Low-coherence interferometry in random media. i. 
theory. J. Opt. Soc. Am. A 17(11), 2024-2033 (2000) 

Bruno, O., Chaubell, J.: One-dimensional inverse scattering problem for optical coherence 
tomography. Inverse Prob. 21, 499-524 (2005) 

Colton, D., Kress, R.: Inverse Acoustic and Electromagnetic Scattering Theory. 2nd edn, In: 
Applied Mathematical Sciences, vol. 93. Springer, Berlin (1998) 


. Dolin, L.S.: A theory of optical coherence tomography. Radiophys. Quantum Electron. 41(10), 


850-873 (1998) 


. Drexler, W., Fujimoto, J.G.: Optical Coherence Tomography. Springer, Berlin (2008) 
. Duan, L., Makita, S., Yamanari, M., Lim, Y., Yasuno, Y.: Monte-carlo-based phase retardation 


estimator for polarization sensitive optical coherence tomography. Opt. Express 19, 16330- 
16345 (2011) 


. Feng, Y., Wang, R.K., Elder, J.B.: Theoretical model of optical coherence tomography for 


system optimization and characterization. J. Opt. Soc. Am. A 20(9), 1792-1803 (2003) 


. Fercher, A.F.: Optical coherence tomography. J. Biomed. Opt. 1(2), 157-173 (1996) 
. Fercher, A.F.: Optical coherence tomography - development, principles, applications. Z. Med. 


Phys. 20, 251-276 (2010) 


. Fercher, A.F., Hitzenberger, C.K.: Optical Coherence Tomography. In: Progress in Optics. 


Elsevier Science B. V., Amsterdam (2002) 


. Fercher, A.F., Drexler, W., Hitzenberger, C.K., Lasser, T.: Optical coherence tomography - 


principles and applications. Rep. Prog. Phys. 66(2), 239-303 (2003) 


. Fercher, A.F., Hitzenberger, C.K., Drexler, W., Kamp, G., Sattmann, H.: In vivo optical 


coherence tomography. Am. J. Ophthalmol. 116, 113-114 (1993) 


. Fercher, A.F., Hitzenberger, C.K., Kamp, G., El Zaiat, S.Y.: Measurement of intraocular 


distances by backscattering spectral interferometry. Opt. Commun. 117, 43-48 (1995) 


. Fercher, A.F., Sander, B., Jorgensen, T.M., Andersen, P.E.: Optical Coherence Tomography. 


In: Encyclopedia of Analytical Chemistry. John Wiley & Sons Ltd., Chichester (2009) 


. Friberg, A.T., Wolf, E.: Angular spectrum representation of scattered electromagnetic fields. 


J. Opt. Soc. Am. 73(1), 26-32 (1983) 


. Hecht, E.: Optics. 4th edn. Addison Wesley, San Francisco (2002) 
. Hellmuth, T.: Contrast and resolution in optical coherence tomography. In: Bigio, LJ., 


Grundfest, W.S., Schneckenburger, H., Svanberg K., Viallet P.M., (eds.) Optical Biopsies and 
Microscopic Techniques. Proceedings of SPIE, vol 2926, pp 228-237 (1997) 


. Hohage, T.: Fast numerical solution of the electromagnetic medium scattering problem and 


applications to the inverse problem. J. Comput. Phys. 214, 224-238 (2006) 

Huang, D., Swanson, E.A., Lin, C.P., Schuman, J.S., Stinson, G., Chang, W., Hee, M.R., Flotte, 
T., Gregory, K., Puliafito, C.A., Fujimoto, J.G.: Optical coherence tomography. Science 
254(5035), 1178-1181 (1991) 

Izatt, J.A., Choma, M.A.: Theory of optical coherence tomography. In: Drexler, W., Fujimoto, 
J.G. (eds.) In: Optical Coherence Tomography, pp. 47-72. Springer, Berlin (2008) 

Kirillin, M., Meglinski, I., Kuzmin, V., Sergeeva, E., Myllylaé, R.: Simulation of optical 
coherence tomography images by monte carlo modeling based on polarization vector approach. 
Opt. Express 18(21), 21714-21724 (2010) 

Kniittel, A., Schork, R., Bocker, D.: Analytical modeling of spatial resolution curves in 
turbid media acquired with optical coherence tomography (oct). In: Cogwell, C.J., Kino, 
G.S., Wilson, T. (eds.) Three- Dimensional Microscopy: Image Acquisition and Processing 
III, Proceedings of SPIE, vol 2655, pp. 258-270 (1996) 

Marks, D.L., Davis, B.J., Boppart, S.A., Carney, P.S.: Partially coherent illumination in 
full-field interferometric synthetic aperture microscopy. J. Opt. Soc. Am. A 26(2), 376-386 
(2009) 

Marks, D.L., Ralston, T.S., Boppart, S.A., Carney, P.S.: Inverse scattering for frequency- 
scanned full-field optical coherence tomography. J. Opt. Soc. Am. A 24(4), 1034-1041 
(2007) 


1204 P. Elbau et al. 


30. 
31 


32; 
33. 


34. 


41. 


42. 


43. 


44. 


45. 


46. 


Orfanidis, S.J.: Electromagnetic Waves and Antennas. Rutgers University Press, NJ (2002) 
Pan, Y., Birngruber, R., Rosperich, J., Engelhardt, R.: Low-coherence optical tomography in 
turbid tissue: theoretical analysis. App. Opt. 34(28), 6564-6574 (1995) 

Podoleanu, A.G.: Optical coherence tomography. Br. J. Radiol. 78, 976-988 (2005) 

Potthast, R.: Integral equation methods in electromagnetic scattering from anisotropic media. 
Math. Methods Appl. Sci. 23, 1145-1159 (2000) 

Ralston, T.S.: Deconvolution methods for mitigation of transverse blurring in optical coherence 
tomography. IEEE Trans. Image Process. 14(9), 1254-1264 (2005) 


. Ralston, T.S., Marks, D.L., Carney, P.S., Boppart, S.A.: Inverse scattering for optical coherence 


tomography. J. Opt. Soc. Am. A 23(5), 1027-1037 (2006) 


. Schmitt, J.M.: Optical coherence tomography (OCT): A review. IEEE J. Quantum Electron. 5, 


1205-1215 (1999) 


. Schmitt, J.M., Kniittel, A.: Model of optical coherence tomography of heterogeneous tissue. 


J. Opt. Soc. Am. A 14(6), 1231-1242 (1997) 


. Schmitt, J.M., Kniittel, A., Bonner, R.F: Measurement of optical properties of biological 


tissues by low-coherence reflectometry. Appl. Opt. 32, 6032-6042 (1993) 


. Schmitt, J.M., Xiang, S.H., Yung, K.M.: Differential absorption imaging with optical 


coherence tomography. J. Opt. Soc. Amer. A 15, 2288-2296 (1998) 


. Smithies, D.J., Lindmo, T., Chen, Z., Nelson, J.S., Milner, T.E.: Signal attenuation and 


localization in optical coherence tomography studied by monte carlo simulation. Phys. Med. 
Biol. 43, 3025-3044 (1998) 

Swanson, E.A., Izatt, J.A., Hee, M.R., Huang, D., Lin, C.P., Schuman, J.S., Puliafito, C.A., 
Fujimoto, J.G.: In vivo retinal imaging by optical coherence tomography. Opt. Lett. 18, 1864— 
1866 (1993) 

Thomsen, J.B., Sander, B., Mogensen, M., Thrane, L., Jgrgensen, T.M., Martini, T., Jemec, 
G.B.E., Andersen, P.E.: Optical coherence tomography: Technique and applications. In: 
Advanced Imaging in Biology and Medicine, pp. 103-129. Springer, Berlin (2009) 

Thrane, L., Yura, H.T., Andersen, P.E.: Analysis of optical coherence tomography systems 
based on the extended huygens - fresnel principle. J. Opt. Soc. Am. A 17(3), 484-490 (2000) 

Tomlins, P.H., Wang, R.K.: Theory, developments and applications of optical coherence 
tomography. J. Phys. D: Appl. Phys. 38, 2519-2535 (2005) 

Turchin, I.V., Sergeeva, E.A., Dolin, L.S., Kamensky, V.A., Shakhova, N.M., Richards Kortum, 
R.: Novel algorithm of processing optical coherence tomography images for differentiation of 
biological tissue pathologies. J. Biomed. Opt. 10(6) 064024, (2005) 

Xu, C., Marks, D.L., Do, M.N., Boppart, S.A.: Separation of absorption and scattering profiles 
in spectroscopic optical coherence tomography using a least-squares algorithm. Opt. Express 
12(20), 4790-4803 (2004) 


Matti Lassas, Mikko Salo, and Gunther Uhlmann 


Contents 
A, DPD ie sryeae salts ocedinw, sare ncdentiondy Sareate a a cdvnesly cise wound enh ai ade teed amie cuidbie ain 1206 
BaCkeroune. co: isi cociis cecal acted signa cesndecums eaten viae oie sieeve ce eatenesneaee ae 1207 
Wave Imaging and Boundary Control Method.................cccece cece ects ee ee eee ees 1207 
‘Travel Times. and Scattering Relation... os...cscc wessiceaeieasteacacesnieabectster neasies 1208 
Ciaveleis ait), Wave BQUuanans: ...cccsicsaiscccsdveasssnedniannaeneeaaseneaaedcet ceaeees 1210 
3 Mathematical Modeling and Analysis. . ...0....00cccascccsusetews nesses dane cesaaesonsenes 1211 
Poundary Control Nem si. cen catia ce cacaiieawa series stheasavnteseiuaubinastatigeis 1211 
Travel Times. and Scattering RelatwoM... 2... ccncs esac teens secdeascs sawn ens nace nee 1234 
Curvelets and: Wave BQnataas.:. 3..4c.2cicegancine ous seanan enone vesvnechindedentie ess. de 1239 
ME. MMMM MN as ican niet ae och alos near. cleae adie HueapAehene tauaenaemsseomss aia ce man ae cate te 1248 
A HOSB MPEMOS. sie siene tag sa RG datta, ohare Sayers Rblen ais al icin a dts bbe wale islSR acalecst ne asva ule heane ete 1248 
WC TeT COS inc ss sha dictinse snweainnes az Sa dp sinned enue madetaieeancaanenebanne ees eeaunemneesl 1248 

Abstract 


This chapter discusses imaging methods related to wave phenomena, and in 
particular, inverse problems for the wave equation will be considered. The first 
part of the chapter explains the boundary control method for determining a 
wave speed of a medium from the response operator, which models boundary 
measurements. The second part discusses the scattering relation and travel times, 
which are different types of boundary data contained in the response operator. 
The third part gives a brief introduction to curvelets in wave imaging for media 
with nonsmooth wave speeds. The focus will be on theoretical results and 
methods. 
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1 Introduction 


This chapter discusses imaging methods related to wave phenomena. Of the 
different types of waves that exist, we will focus on acoustic waves and problems 
which can be modeled by the acoustic wave equation. In the simplest case, this is 
the second-order linear hyperbolic equation 


d-u(x, t) — c(x) Au(x, t) = 0 


for a sound speed c(x). This equation can be considered as a model for other 
hyperbolic equations, and the methods presented here can in some cases be extended 
to study wave phenomena in other fields such as electromagnetism or elasticity. 

We will mostly be interested in inverse problems for the wave equation. In these 
problems, one has access to certain measurements of waves (the solutions uv) on 
the surface of a medium, and one would like to determine material parameters (the 
sound speed c) of the interior of the medium from these boundary measurements. 
A typical field where such problems arise is seismic imaging, where one wishes to 
determine the interior structure of Earth by making various measurements of waves 
at the surface. We will not describe seismic imaging applications in more detail here, 
since they are discussed elsewhere in this volume. 

Another feature in this chapter is that we will consistently consider anisotropic 
materials, where the sound speed depends on the direction of propagation. This 
means that the scalar sound speed c(x), where x = (x!,x’,...,x") € QC R’, 
is replaced by a positive definite symmetric matrix (g/* (x))" pan and the wave 
equation becomes 


Pad au 
d?u(x, t) = > g(x) Fay, t) =0. 


jok=1 


Anisotropic materials appear frequently in applications such as in seismic imaging. 

It will be convenient to interpret the anisotropic sound speed (g/*) as the 
inverse of a Riemannian metric, thus modeling the medium as a Riemannian 
manifold. The benefits of such an approach are twofold. First, the well-established 
methods of Riemannian geometry become available to study the problems, and 
second, this provides an efficient way of dealing with the invariance under changes 
of coordinates present in many anisotropic wave imaging problems. The second 
point means that in inverse problems in anisotropic media, one can often only 
expect to recover the matrix (g/*) up to a change of coordinates given by some 
diffeomorphism. In practice, this ambiguity could be removed by some a priori 
knowledge of the medium properties (such as the medium being in fact isotropic, 
see section “From Boundary Distance Functions to Riemannian Metric’). 
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2 Background 


This chapter contains three parts which discuss different topics related to wave 
imaging. The first part considers the inverse problem of determining a sound speed 
in a wave equation from the response operator, also known as the hyperbolic 
Dirichlet-to-Neumann map, by using the boundary control method; see [5, 7, 42]. 
The second part considers other types of boundary measurements of waves, namely, 
the scattering relation and boundary distance function, and discusses corresponding 
inverse problems. The third part is somewhat different in nature and does not 
consider any inverse problems but rather gives an introduction to the use of curvelet 
decompositions in wave imaging for nonsmooth sound speeds. We briefly describe 
these three topics. 


Wave Imaging and Boundary Control Method 


Let us consider an isotropic wave equation. Let 2 C R"” be an open, bounded 
set with smooth boundary 0Q, and let c(x) be a scalar-valued positive function in 
C™() modeling the wave speed in (2. First, we consider the wave equation 


d?u(x,t) —c(x)?Au(x,t) =0 inQx Ry, (1) 
ulr=0 =0, Ur|r=0 = 0, 


e(x)"t Ogu = f(x,t) indQx Ry, 


where 0, denotes the Euclidean normal derivative and n is the unit interior normal. 
We denote by uf = u/ (x,t) the solution of (1) corresponding to the boundary 
source term /f. 

Let us assume that the domain 82 C R" is known. The inverse problem is to 
reconstruct the wave speed c(x) when we are given the set 


{(f laaxo.2r), ul |aaxo2ry) : f € C§°(8Q x Ri)}, 


that is, the Cauchy data of solutions corresponding to all possible boundary sources 
f € C§P(dQ x Ry), T € (0, oo]. If T = ov, then this data is equivalent to the 
response operator 


Ag: f Buf lpaxr,, (2) 


which is also called the nonstationary Neumann-to-Dirichlet map. Physically, Ag f 
describes the measurement of the medium response to any applied boundary source 
Ff, and it is equivalent to various physical measurements. For instance, measuring 
how much energy is needed to force the boundary value c(x)~"t!dnuljaxr 4 to 
be equal to any given boundary value f € Cj°(dQ2 x R,) is equivalent to 
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measuring the map Ag on 0Q x R+; see [42,44]. Measuring Ag is also equivalent 
to measuring the corresponding Neumann-to-Dirichlet map for the heat or the 
Schrédinger equations or measuring the eigenvalues and the boundary values of 
the normalized eigenfunctions of the elliptic operator —c(x)? A; see [44]. 

The inverse problems for the wave equation and the equivalent inverse problems 
for the heat or the Schrédinger equations go back to works of M. Krein at the 
end of the 1950s, who used the causality principle in dealing with the one- 
dimensional inverse problem for an inhomogeneous string, u;; — c?(x)uxx = 0; 
see, for example, [46]. In his works, causality was transformed into analyticity of 
the Fourier transform of the solution. A more straightforward hyperbolic version 
of the method was suggested by A. Blagovestchenskii at the end of 1960s to 
1970s [12, 13]. The multidimensional case was studied by M. Belishev [4] in the 
late 1980s who understood the role of the PDE control for these problems and 
developed the boundary control method for hyperbolic inverse problems in domains 
of Euclidean space. Of crucial importance for the boundary control method was 
the result of D. Tataru in 1995 [77,79] concerning a Holmgren-type uniqueness 
theorem for nonanalytic coefficients. The boundary control method was extended to 
the anisotropic case by M. Belishev and Y. Kurylev [7]. The geometric version of 
the boundary control method which we consider in this chapter was developed in 
[7,41,42,47]. We will consider the inverse problem in the more general setting of an 
anisotropic wave equation in an unbounded domain or on a non-compact manifold. 
These problems have been studied in detail in [39, 43] also in the case when the 
measurements are done only on a part of the boundary. In this paper we present a 
simplified construction method applicable for non-compact manifolds in the case 
when measurements are done on the whole boundary. We demonstrate these results 
in the case when we have an isotropic wave speed c(x) in a bounded domain of 
Euclidean space. For this, we use the fact that in the Euclidean space, the only 
conformal deformation of a compact domain fixing the boundary is the identity 
map. This implies that after the abstract manifold structure (M, g) corresponding 
to the wave speed c(x) in a given domain Q is constructed, we can construct in an 
explicit way the embedding of the manifold M to the domain Q and determine c(x) 
at each point x € Q. We note on the history of this result that using Tataru’s unique 
continuation result [77], Theorem 2 concerning this case can be proven directly 
using the boundary control method developed for domains in Euclidean space in [4]. 

The reconstruction of non-compact manifolds has been considered also in [11, 
27] with different kind of data, using iterated time reversal for solutions of the wave 
equation. We note that the boundary control method can be generalized also for 
Maxwell and Dirac equations under appropriate geometric conditions [50,51], and 
its stability has been analyzed in [1,45]. 


Travel Times and Scattering Relation 


The problem considered in the previous section of recovering a sound speed from 
the response operator is highly overdetermined in dimensions n > 2. The Schwartz 
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kernel of the response operator depends on 2n variables, and the sound speed c 
depends on n variables. 

In section “Travel Times and Scattering Relation,’ we will show that other 
types of boundary measurements in wave imaging can be directly obtained from 
the response operator. One such measurement is the boundary distance function, a 
function of 2” — 2 variables, which measures the travel times of shortest geodesics 
between boundary points. The problem of determining a sound speed from the travel 
times of shortest geodesics is the inverse kinematic problem. The more general 
problem of determining a Riemannian metric (corresponding to an anisotropic 
sound speed) up to isometry from the boundary distance function is the boundary 
rigidity problem. The problem is formally determined ifn = 2 but overdetermined 
forn > 3. 

This problem arose in geophysics in an attempt to determine the inner structure 
of the Earth by measuring the travel times of seismic waves. It goes back to 
Herglotz [37] and Wiechert and Zoeppritz [84] who considered the case of a radial 
metric conformal to the Euclidean metric. Although the emphasis has been in 
the case that the medium is isotropic, the anisotropic case has been of interest in 
geophysics since the Earth is anisotropic. It has been found that even the inner core 
of the Earth exhibits anisotropic behavior [24]. 

To give a proper definition of the boundary distance function, we will consider 
a bounded domain 22 C R” with smooth boundary to be equipped with a 
Riemannian metric g, that is, a family of positive definite symmetric matrices 


g(x) = (jx (2) eat depending smoothly on x € Q. The length of a smooth 


curve y : [a,b] > Q is defined to be 


1/2 


b n 
L,(y) = / Sgro (ope) | — at. 


jk=1 


The distance function d,(x, y) for x,y € Q is the infimum of the lengths of all 
piecewise smooth curves in Q joining x and y. The boundary distance function is 
d,(x, y) forx,y € dQ. 

In the boundary rigidity problem, one would like to determine a Riemannian 
metric g from the boundary distance function d,. In fact, since dg = dy, for any 
diffeomorphism y : Q —> Q which fixes each boundary point, we are looking 
to recover from dy, the metric g up to such a diffeomorphism. Here, w*g(y) = 
Dw(y)'g(W(y)) DW(y) is the pullback of g by y. 

It is easy to give counterexamples showing that this cannot be done in general; 
consider, for instance, the closed hemisphere, where boundary distances are given 
by boundary arcs so making the metric larger in the interior does not change dy. 
Michel [55] conjectured that a simple metric g is uniquely determined, up to an 
action of a diffeomorphism fixing the boundary, by the boundary distance function 
d,(x, y) known for all x and y on dQ. A metric is called simple if for any two 
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points in Q, there is a unique length minimizing geodesic joining them, and if the 
boundary is strictly convex. 

The conjecture of Michel has been proved for two-dimensional simple manifolds 
[60]. In higher dimensions, it is open, but several partial results are known, including 
the recent results of Burago and Ivanov for metrics close to Euclidean [15] and close 
to hyperbolic [16] (see the survey [40]). Earlier and related works include results 
for simple metrics conformal to each other [8, 10, 26, 56-58], for flat metrics [34], 
for locally symmetric spaces of negative curvature [9], for two-dimensional simple 
metrics with negative curvature [25,59], a local result [70], a semiglobal solvability 
result [54], and a result for generic simple metrics [71]. 

In case the metric is not simple, instead of the boundary distance function, one 
can consider the more general scattering relation which encodes, for any geodesic 
starting and ending at the boundary, the start point and direction, the end point and 
direction, and the length of the geodesic. We will see in section “Travel Times and 
Scattering Relation” that also this information can be determined directly from 
the response operator. If the metric is simple, then the scattering relation and 
boundary distance function are equivalent, and either one is determined by the 
other. 

The lens rigidity problem is to determine a metric up to isometry from the 
scattering relation. There are counterexamples of manifolds which are trapping, and 
the conjecture is that on a nontrapping manifold the metric is determined by the 
scattering relation up to isometry. We refer to [72] and the references therein for 
known results on this problem. 


Curvelets and Wave Equations 


In section “Curvelets and Wave Equations,” we describe an alternative approach to 
the analysis of solutions of wave equations, based on a decomposition of functions 
into basic elements called curvelets or wave packets. This approach also works 
for wave speeds of limited smoothness unlike some of the approaches presented 
earlier. Furthermore, the curvelet decomposition yields efficient representations of 
functions containing sharp wave fronts along curves or surfaces, thus providing 
a common framework for representing such data and analyzing wave phenomena 
and imaging operators. Curvelets and related methods have been proposed as 
computational tools for wave imaging, and the numerical aspects of the theory are a 
subject of ongoing research. 

A curvelet decomposition was introduced by Smith [67] to construct a solution 
operator for the wave equation with C'! sound speed and to prove Strichartz 
estimates for such equations. This started a body of research on L? estimates for 
low-regularity wave equations based on curvelet-type methods; see, for instance, 
Tataru [80-82], Smith [68], and Smith and Sogge [69]. Curvelet decompositions 
have their roots in harmonic analysis and the theory of Fourier integral operators, 
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where relevant works include Cérdoba and Fefferman [23] and Seeger et al. [65] 
(see also Stein [73]). 

In arather different direction, curvelet decompositions came up in image analysis 
as an optimally sparse way of representing images with C? edges; see Candés 
and Donoho [20] (the name “curvelet’” was introduced in [19]). The property 
that curvelets yield sparse representations for wave propagators was studied in 
Candés and Demanet [17, 18]. Numerical aspects of curvelet-type methods in wave 
computation are discussed in [21,30]. Finally, both theoretical and practical aspects 
of curvelet methods related to certain seismic imaging applications are studied in 
[2, 14,29, 31,64]. 


3 Mathematical Modeling and Analysis 
Boundary Control Method 


Inverse Problems on Riemannian Manifolds 

Let 2 Cc R" be an open, bounded set with smooth boundary dQ and let c(x) be 
a scalar-valued positive function in C°(Q), modeling the wave speed in Q. We 
consider the closure Q as a differentiable manifold M with a smooth, nonempty 
boundary. We consider also a more general case, and allow (M, g) to be a possibly 
non-compact, complete manifold with boundary. This means that the manifold 
contains its boundary dM and M is complete with metric d, defined below. 
Moreover, near each point x € M, there are coordinates (U, X), where U C M is 
a neighborhood of x and X : U > R" if x is an interior point, or X : U > R""! x 
[0, co) if x is a boundary point such that for any coordinate neighborhoods (U, X) 
and (U, X), the transition functions X¥ o X~-! : X(UNU) > X(UNU) are C™- 
smooth. Note that all compact Riemannian manifolds are complete according to this 
definition. Usually we denote the components of X by X(y) = (x'(y),...,x"()). 

Let u be the solution of the wave equation 


uy,(x,t)+ Au(x,t) =O in MxR,, (3) 
uli=0 = 9, us |r=0 = 0, 
By Ulam xr, = f 
Here, f € Cp°(0M x R+) is a real-valued function, and A = A(x, D) is an elliptic 
partial differential operator of the form 


n 


Av =— )* way IgQ@-? 


jk=l 


is ty 0 
(uiiele" ree) + g(x)v(x), (4) 


Oxi 


where g/*(x) is a smooth, symmetric, real, positive definite matrix, |g| = 
det(g/*(x))™', and (x) > O and q(x) are smooth real-valued functions. On 
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existence and properties of the solutions of Eq. (3), see [52]. The inverse of the 


matrix (g/*(x))" _,, denoted (¢ 5, (x " _, defines a Riemannian metric on M. 
& jk=l gi jk=l 


The tangent space of M at x is denoted by 7).M, and it consists of vectors p 
which in local coordinates (U, X), X(y) = Cae’) re (y)) are written as 
p= Fy pee. Similarly, the cotangent space T*M of M at x consists of 
covectors which are written in the local coordinates as § = )°y_, &.dx*. The 
inner product which g determines in the cotangent space T*M of M at the point 


x is denoted by (&,)¢ = g(&,) = Sen gl (x)Eink for &,n € T*M. We 
use the same notation for the inner product at the tangent space 7, M, that is, 


(D.9)¢ = 8(7.9) = Lipa Bix (~) p'g* for p,g € T,M. 
The metric defines a distance function, which we call also the travel time 
function, 


1 
d,(x, y) = inf |_|, w= f (A, 10(s), Os 40(s))¥/7ds, 
0 


where || denotes the length of the path jz, and the infimum is taken over all 
piecewise C'-smooth paths jz : [0, 1] > M with (0) = x and p(1) = y. 
We define the space L?(M, dV,,) with inner product 


(1 )2anayy =f wesv(a) dV, (0, 


where dV, = p(x)|g¢(x)|'/2dx!dx?...dx". By the above assumptions, A is 
formally self-adjoint, that is, 


(Au, v) panavy = (Ar) 2arav,) foru,v € Ceo(M"™). 
Furthermore, let 
By nv = —d,v + nV, 


where 7 : JM — R is a smooth function and 


n 


0 
dv = D7) (xa! (xv sv), 


k=l 


where v(x) = (v1, ¥2,..., Vm) is the interior conormal vector field of 0M, satisfying 
yi g/*v;& = 0 for all cotangent vectors of the boundary, § € T*(0M). We 
assume that v is normalized, so that ee gik vj; vz, = 1. If M is compact, then 
the operator A in the domain D(A) = {v € H?(M) : d,v\ay = 0%, where 
H*(M) denotes the Sobolev spaces on M, is an unbounded self-adjoint operator 
in L?(M,dV,,). 
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An important example is the operator 
Ag = —c7(x)A + q(x) (5) 


on a bounded smooth domain Q C R” with d,v = c(x)~"*!dgv, where dav is the 
Euclidean normal derivative of v. 
We denote the solutions of (3) by 


u(x,t) = uf (x, ft). 


For the initial boundary value problem (3), we define the nonstationary Robin-to- 
Dirichlet map or the response operator A by 


Af =u lamxr,- (6) 


The finite time response operator A’ corresponding to the finite observation time 
T > Ois given by 


AT f =u! |amxo.r): (7) 


For any set B C 0M x R4, we denote L7(B) = {f € L?(0M x Ri): 
supp(f) C B}. This means that we identify the functions and their zero contin- 
uations. 

By [78], the map A” can be extended to bounded linear map A? : L7(B) > 
H'/3(9M x (0,T)) when B Cc 0M x (0,T) is compact. Here, H*(0M x (0, T)) 
denotes the Sobolev space on 0M x (0, 7). Below we consider A? also as a linear 
operator AT : L2,,(M x (0,T)) > L*(4M x (0,T)), where L2,,(0M x (0, T)) 
denotes the compactly supported functions in L* (0M x (0, T)). 

For t > 0 and a relatively compact open set T C 0M, let 


M(T,t)={x eM : dg(x,T) < ft}. (8) 
This set is called the domain of influence of I’ at time f. 

When I C 0M is an open relatively compact set and f € Cj°(T x R4), it 
follows from finite speed of wave propagation (see, e.g., [38]) that the wave uf (t) = 
u/ (-,) is supported in the domain M(T, f), that is, 

ul (t) € L?(M(I,1t)) = {v € L?(M): supp(v) c M(T,1)}. (9) 

We will consider the boundary of the manifold 0M with the metric gay = i*g 


inherited from the embedding 1 : dM — M. We assume that we are given the 
boundary data, that is, the collection 


(0M, gam) and A, (10) 
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where (0M, gay) is considered as a smooth Riemannian manifold with a known 
differentiable and metric structure and A is the nonstationary Robin-to-Dirichlet 
map given in (6). 

Our goal is to reconstruct the isometry type of the Riemannian manifold (M, g), 
that is, a Riemannian manifold which is isometric to the manifold (M, g). This is 
often stated by saying that we reconstruct (M, g) up to an isometry. Our next goal 
is to prove the following result: 


Theorem 1. Let (M,g) to be a smooth, complete Riemannian manifold with a 
nonempty boundary. Assume that we are given the boundary data (10). Then it is 
possible to determine the isometry type of manifold (M, g). 


From Boundary Distance Functions to Riemannian Metric 

In order to reconstruct (M,g), we use a special representation, the boundary 
distance representation, R(M), of M and later show that the boundary data (10) 
determine R(M). We consider next the (possibly unbounded) continuous functions 
h: C(0M) — R. Let us choose a specific point Oo € 0M and a constant Co > 0 
and using these, endow C(0M) with the metric 


dc (hy, hz) = |hy(Qo) — h2(Qo)| + ey min(Co, |/1(z) — h2(z)|). (11) 
z€0M 


Consider a map R: M > C(dM), 
R(x) =rx(-)s tx(Z) = de(x,2), < € OM, (12) 


that is, r,(-) is the distance function from x € M to the points on dM. The image 
R(M) Cc C(0M) of R is called the boundary distance representation of M. The set 
R(M) is a metric space with the distance inherited from C(0M) which we denote 
by dc, too. The map R, due to the triangular inequality, is Lipschitz, 


Ac(rx, Ty) < 2dg(x, y). (13) 


We note that when M is compact and Cy = diam (M), the metric dc : C(0M) > R 
is a norm which is equivalent to the standard norm || f |lo = maxyeam | f(x)| of 
C(dM). 

We will see below that the map R : M — R(M) C C(0M) is an embed- 
ding. Many results of differential geometry, such as Whitney or Nash embedding 
theorems, concern the question how an abstract manifold can be embedded to some 
simple space such as a higher dimensional Euclidean space. In the inverse problem, 
we need to construct a “copy” of the unknown manifold in some known space, and 
as we assume that the boundary is given, we do this by embedding the manifold M 
to the known, although infinite dimensional function space C(dM ). 

Next we recall some basic definitions on Riemannian manifolds; see, for 
example, [22] for an extensive treatment. A path w : [a,b] — N is called a 
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geodesic if, for any c € [a,b], there is ¢ > 0 such that if s,t € [a,b] such that 
c-é<s <t<c+e, the path p(s, t]) is a shortest path between its endpoints, 
that is, 


|([s.t])| = de(u(s), wD). 


In the future, we will denote a geodesic path jz by y and parameterize y with its 
arclength s, so that |w([s1, 52])| = de(u(s1), [L(S2)). Let x(s), 


x) (0 Bde dO), 


be the representation of the geodesic y in local coordinates (U, X). In the interior 
of the manifold, that is, for U C M* the path x(s) satisfies the second-order 
differential equations 


1% “() _ yes (s) dx/(s) 
Se 7. (14) 


ds? ds ~ 
i,j=l 


where ry are the Christoffel symbols, given in local coordinates by the formula 


n Og; ; 
rh) = Do 58 (# Sin (x) 4 Bie (gy — B(x )). 


p=1 


Let y € M andé € T,M bea unit vector satisfying the condition g(é, v(y)) >0 
in the case when y € 0M. Then, we can consider the solution of the initial value 
problem for the differential equation (14) with the initial data 


d 
x@)=y, =O) =6. 


This initial value problem has a unique solution x(s) on an interval [0, so(y, €)) 
such that so(y,&) > 0 is the smallest value so > O for which x(s9) € 0M, or 
So(y, €) = 00 in case no such so exists. We will denote x(s) = y,¢(s) and say that 
the geodesic is a normal geodesic starting at y if y € OM and & = v(y). 


Example 1. In the case when (M, g) is such a compact manifold that all geodesics 
are the shortest curves between their endpoints and all geodesics can be continued 
to geodesics that hit the boundary, we can see that the metric spaces (M, d,) and 
(R(M), ||- loo) are isometric. Indeed, for any two points x,y € M, there is a 
geodesic y from x to a boundary point z, which is a continuation of the geodesic 
from x to y. As in the considered case the geodesics are distance minimizing curves, 
we see that 
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rx(z) —Try(z) = de(x, Z) — dg(y,2) = de(x,y), 


and thus ||rx — ry loo = dg(x, y). Combining this with the triangular inequality, we 
see that ||rx — ry|loo = dg(x, y) for x,y € M and R is isometry of (M, d,) and 
(R(M), |I- Ilo). 

Notice that when even M is a compact manifold, the metric spaces (M, d,) and 
(R(M), || - ||o) are not always isometric. As an example, consider a unit sphere in 
IR? with a small circular hole near the South pole of, say, diameter ¢. Then, for any 
x, y on the equator and z € 0M, m/2—e€ < r,(z) < m/2and2/2—« < r,(z) < w/2. 
Then dc (rx, ry) < €, while dg(x, y) may be equal to z. 

Next, we introduce the boundary normal coordinates on M. For a normal 
geodesic y,,,(s) starting from z € 0M consider dg (yz1(s), 0M). For small s, 


dg(yzy(s), 0M) = S$, (15) 


and z is the unique nearest point to y,,(s) on 0M. Let t(z) € (0, co] be the largest 
value for which (15) is valid for all s € [0, t(z)]. Then for s > t(z), 


dg(Vzv(s),M) <s, 


and z is no more the nearest boundary point for y,,(s). The function t(z) € C(dM) 
is called the cut locus distance function, and the set 


w = {yzy(t(z)) € M: z € OM, and t(z) < o0}, (16) 


is the cut locus of M with respect to 0M. The set w is a closed subset of M having 
zero measure. In particular, M/\q@ is dense in M. In the remaining domain M\a, 
we can use the coordinates 


x+> (2(x),t(x)), (17) 


where z(x) € 0M is the unique nearest point to x and t(x) = dg(x, dM). (Strictly 
speaking, one also has to use some local coordinates of the boundary, y : z > 
(v'(z),..., ¥~)(z)) and define that 


x (y(z(x)),t00)) = (vex)... xP (ex), 4%) ER", (18) 


are the boundary normal coordinates.) Using these coordinates, we show that R : 
M — C(dM) is an embedding. The result of Lemma | is considered in detail for 
compact manifolds in [42]. 
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Lemma 1. Let (M,d,) be the metric space corresponding to a complete Rie- 
mannian manifold (M, g) with a nonempty boundary. The map R : (M,d,) > 
(R(M), dc) is a homeomorphism. Moreover, given R(M) as a subset of C(dM), 
it is possible to construct a distance function dr on R(M) that makes the metric 
space (R(M), dp) isometric to (M, dg). 


Proof. We start by proving that R is a homeomorphism. Recall the following simple 
result from topology: 

Assume that X and Y are Hausdorff spaces, X is compact, and F : X — Y isa 
continuous, bijective map from X to Y. Then F : X — Y is a homeomorphism. 

Let us next extend this principle. Assume that (X, dy) and (Y, dy) are metric 
spaces and let X¥; C X, j € Z4 be compact sets such that Ujez, X; = X. 
Assume that F : X — Y is acontinuous, bijective map. Moreover, let Y; = F(X;) 
and assume that there is a point p € Y such that 


a; 


j= inf dy(y,p)—> wasj > ow. (19) 


i 
yeY\Y; 
Then by the above, the maps F' : Ui aX ja Ui a1 Y; are homeomorphisms for all 
n € Z,. Next, consider a sequence yz € Y such that y, > yin Y ask > oo. 
By removing first elements of the sequence (y;,)?2, if needed, we can assume that 
dy (yz, y) < 1. Let now N € Zy be such that for 7 > N, we have aj > b := 
dy(y, p) + 1. Then yx € Les Y; and as the map F : L_, X; > U*_, Y; is 


j=l j=l 
a homeomorphism, we see that F~!(y,) > F7!(y) in X as k — oo. This shows 
that F~! : Y — X is continuous, and thus F : X > Y isa homeomorphism. 


By definition, R : M — R(M) is surjective and, by (13), continuous. In order to 
prove the injectivity, assume the contrary, that is, 7,(-) = ry(-) but x # y. Denote 
by zo any point where 


min rx (z) = rx (Zo). 
Then 


dg(x,dM) = ami rx (z) = rx (Zo) (20) 


II 


ry (zo) = ry(z) = dg(y, 0M), 


and z € 0M is a nearest boundary point to x. Let 4x be the shortest path from z 
to x. Then, the path j, is a geodesic from x to zo which intersects 0M first time 
at z. By using the first variation on length formula, we see that jz, has to hit to 
z normally; see [22]. The same considerations are true for the point y with the 
same point Zp. Thus, both x and y lie on the normal geodesic y,,,(s) to 0M. As the 
geodesics are unique solutions of a system of ordinary differential equations (the 
Hamilton—Jacobi equation (14)), they are uniquely determined by their initial points 
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and directions, that is, the geodesics are non-branching. Thus, we see that 
X = Yzo(So) = y, 
where so = rx(Zo) = ry(zo). Hence, R: M — C(dM) is injective. 
Next, we consider the condition (19) for R : M — R(M). Let z € M and 


consider closed sets X; = {x € M: dc (R(x), R(z)) < j}, 7 € Z4. Then for 
x € X;, we have by definition (11) of the metric dc that 


d,(x, Qo) < j +d¢(z, Qo), 


implying that the sets X;, 7 € Z+ are compact. Clearly, J jeLy X; = X. Let next 
Y; = R(X;) C Y = R(M) and p = R(Qo) € R(M). Then for ry € Y\Y;, we 
have 


V 


dc (rx, P) 2 rx(Qo) — p(Qo) = de(x, Qo) 


J —dg(z, Qo) — Cy > co as j > 00 


IV 


and thus the condition (19) is satisfied. As R : M — R(M) is a continuous, 
bijective map, it implies that R : M — R(M) is a homeomorphism. 

Next we introduce a differentiable structure and a metric tensor, gr, on R(M) to 
have an isometric diffeomorphism 


R: (M,g) > (R(M), gp). (21) 


Such structures clearly exists — the map R pushes the differentiable structure of M 
and the metric g to some differentiable structure on R(M) and the metric gr := 
R,g which makes the map (21) an isometric diffeomorphism. Next we construct 
these coordinates and the metric tensor in those on R(M) using the fact that R(/) 
is known as a subset of C(0M). 

We will start by construction of the differentiable and metric structures on 
R(M)\R(q@), where w is the cut locus of M with respect to 0M. First, we show 
that we can identify in the set R(M) all the elements of the form r = r, € R(M) 
where x € M\w. To do this, we observe that r = r, with x = y,,(s), 5 < t(z) if 
and only if: 


1. r(-) has a unique global minimum at some point z € 0M; 

2. There is 7 € R(M) having a unique global minimum at the same z and r(z) < 
r(z). This is equivalent to saying that there is y with r,(- ) having a unique global 
minimum at the same z and r;(z) < ry(z). 


Thus, we can find R(M \w) by choosing all those r € R(M) for which the above 
conditions (1) and (2) are valid. 
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Next, we choose a differentiable structure on R(M \w) which makes the map R : 
M\o — R(M\qo) a diffeomorphism. This can be done by introducing coordinates 
near each r? € R(M\q). Ina sufficiently small neighborhood W Cc R(M) of r° 
the coordinates 


re (Y(r), T(r)) = (v(oremin.cay) min r) 


are well defined. These coordinates have the property that the map x +t 
(Y (rx), T(rx)) coincides with the boundary normal coordinates (17) and (18). 
When we choose the differential structure on R(M\w) that corresponds to these 
coordinates, the map 


R:M\o > R(M\o) 


is a diffeomorphism. 

Next we construct the metric gr on R(M). Let r° € R(M\q). As above, in 
a sufficiently small neighborhood W Cc R(M) of r°, there are coordinates r 
X(r) := (Y(r), T(r)) that correspond to the boundary normal coordinates. Let 
(y°, t°) = X(r°). We consider next the evaluation function 


Ky: WR, Ky(r) = r(w), 


where w € 0M. The inverse of X : W — R" is well defined in a neighborhood 
U CR" of (y°, t°), and thus we can define the function 


Ey, = K,oX7!:U—>R 
that satisfies 


Ew(y,t) = dg (w, Ve(y).v(y)(t)) » (y,t) €U, (22) 


where y-,y),v() (t) is the normal geodesic starting from the boundary point z(y) with 
coordinates y = (y!,..., y"~') and v(y) is the interior unit normal vector at y. 

Let now gr = Rxg be the push-forward of g to R(M\qw). We denote its 
representation in X -coordinates by gj; (y,t). Since X corresponds to the boundary 
normal coordinates, the metric tensor satisfies 


fam = 1, Zam = 07 a=1,....n—1. 


Consider the function E,,(y,t) as a function of (y,f) with a fixed w. Then its 
differential, dE\, at point (y,t) defines a covector in Thay (U ) = R’. Since the 
gradient of a distance function is a unit vector field, we see from (22) that 
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n—-1 


9 2 
IdE Wy. 0g 2 Gao) + DS (eny"0.0) 


a,p=1 


dEy, _. dEw 
, y= 1. 
Bye (y,t) yh (y,t) 


Let us next fix a point (y°,t°) € U. Varying the point w € 0M, we obtain a 


set of covectors dE,,(y°, t°) in the unit ball of (Teo 0)U. ,g in) which contains an 


open neighborhood of (0, ..., 0, 1). This determines uniquely the tensor g/*(y®, f°). 
Thus, we can construct the metric tensor in the boundary normal coordinates at 
arbitrary r € R(M\w). This means that we can find the metric gr on R(M\w) 
when R(M) is given. 

To complete the reconstruction, we need to find the differentiable structure and 
the metric tensor near R(w). Let r € R(w) and x © M'™ be such a point that 
r© = ro = R(x), Let zo be some of the closest points of dM to the point 
x Then there are points z;,...,Z,—1 on 0M, given by Zj = LUzp,6; (So), where 
Hzy,9;(S) are geodesics of (0M, gay) and ),...,,—; are orthonormal vectors of 
T;,(0M) with respect to metric gay and so > 0 is sufficiently small, so that the 
distance functions y bt» dg (zi, ¥)s i = 0,1,2,...,n — 1 form local coordinates 


yR (d 2 (Zi y))ro9 on M in some neighborhood of the point x (we omit here the 
proof which can be found in [42, Lemma 2.14]). 

Let now W C R(M) be a neighborhood of r and let 7 € W. Moreover, let 
V = R'(W) C M and xX = R™'(F) € V. Let us next consider arbitrary points 
Z1,+++s2n-1 on OM. Our aim is to verify whether the functions x + X'(x) = 
d,(x,z:),i =0,1,...,n — 1 form smooth coordinates in V. As M \w is dense on 
M and we have found topological structure of R(M) and constructed the metric 
gr on R(M\w), we can choose r‘/) € R(M\q) such that limj— oor) = 7 in 
R(M). Let x) € M\q be the points for which rY) = R(x). Now the function 
xR (x ' (x))" defines smooth coordinates near x if and only if for functions 
Z'(r) = K;,(r), we have 


Jim det ((ge(@Zi(r),dZ'())) 25) lravd (23) 


= we det ((g(4X' (x), ax! (x) 70) lrax) # O. 


Thus, for all 7 € W we can verify for any points z,,...,Z,—1 € 0M whether the 
condition (23) is valid or not and this condition is valid for all 7 € W if and only if 
the functions x BH X! (x) = d,(x,z),i =0,1,...,2—1 form smooth coordinates 
in V. Moreover, by the above reasoning, we know that any r € R(w) has some 
neighborhood W and some points z;,...,Z,—1 € 0M for which the condition (23) 
is valid for all 7 € W. By choosing such points, we find also near r € (@) 
smooth coordinates r 1 (Ze). which make the map R: M > R(M)a 


diffeomorphism near x. 
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Summarizing, we have constructed differentiable structure (i.e., local coordi- 
nates) on the whole set R(M), and this differentiable structure makes the map 
R:M — R(M) a diffeomorphism. Moreover, since the metric gre = Ryg isa 
smooth tensor, and we have found it in a dense subset R(M\w) of R(M), we can 
continue it in the local coordinates. This gives us the metric gr on the whole R(M), 
which makes the map R : M — R(M) an isometric diffeomorphism. | 


In the above proof, the reconstruction of the metric tensor in the boundary normal 
coordinates can be considered as finding the image of the metric in the travel time 
coordinates. 

Let us next consider the case when we have an unknown isotropic wave speed 
c(x) in a bounded domain Q C R”. We will assume that we are given the set 
Q and an abstract Riemannian manifold (M, g), which is isometric to Q endowed 
with its travel time metric corresponding to the wave speed c(x). Also, we assume 
that we are given a map y : OQ — 0M, which gives the correspondence between 
the boundary points of Q and M. Next we show that it is then possible to find an 
embedding from the manifold M to Q which gives us the wave speed c(x) at each 
point x € Q. This construction is presented in detail, e.g., in [42]. 

For this end, we need first to reconstruct a function o on M which corresponds 
to the function c(x)? on Q. This is done on the following lemma. 


Lemma 2. Assume we are given a Riemannian manifold (M, g) such that there 
exists an open set {2 C R" and an isometry WV : (Q, (o(x)) '5:) — (M, g) and 
a function a on M such that a(W(x)) = o0(x). Then knowing the Riemannian 
manifold (M, g), the restriction y = Wag : OQ — OM, and the boundary value 
Olag, we can determine the function a. 


Proof. First, observe that we are given the boundary value a@lay of a(W(x)) = 
o(x). By assumption, the metric g on M is conformally Euclidean, that is, the 
metric tensor, in some coordinates, has the form gjx(x) = o(x)~!5jx, where 
o(x) > 0. Hence, the function 6B = 5 In(w), when m = 2, and B = oe 2)/4 | 
when n > 3, satisfies the so-called scalar curvature equation 


AgB —kg =0 (n= 2), (24) 


4(n —1 
se gf — ke = 0 (n > 3), (25) 


where kg is the scalar curvature of (M, g), 


k(x) = SY) gl Ox)Ri (x) 


kjl=l 


where R’ ,, 18 the curvature tensor given in terms of the Christoffel symbols as 
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0. 0 _. it : : 
Rigi) = FeV) — saTy) + DO (TH @ME-@) — TOIT). 
r=1 

The idea of these equations is that if 6 satisfies, for example, Eq. (25) in the case 
m > 3, then the metric B*/""—?) g has zero scalar curvature. Together with boundary 
data (10) being given, we obtain Dirichlet boundary value problem for 6 in M. 

Clearly, Dirichlet problem for Eq. (24) has a unique solution that gives a when 
n = 2. In the case n > 3, to show that this boundary value problem has a 
unique solution, it is necessary to check that 0 is not an eigenvalue of the operator 
ast A, — kg with Dirichlet boundary condition. Now, the function B = a'"~?)/4 
is a positive solution of the Dirichlet problem for Eq. (25) with boundary condition 
Bay= at" |; yy: Assume that there is another possible solution of this problem, 


B=vB, v>0, viay =1. (26) 


Then both (M pre g) and (m ; B4/ ee) s) have zero scalar curvatures. Denot- 


ing g, = BYOAg, gw = pl ("—2) ¢. we obtain that v should satisfy the scalar 
curvature equation 


4(n — 1) 
ag AY — ke = 0. 


Here, we have kz, = O as g; has vanishing scalar curvature. Together with boundary 
condition (26), this equation implies that v = 1, that is, B = B . This immediately 
yields that 0 is not the eigenvalue of the Dirichlet operator (25) because, otherwise, 
we could obtain a positive solution B = B + coWo, where Wo is the Dirichlet 
eigenfunction, corresponding to zero eigenvalue, and |co| is sufficiently small. Thus, 
B, and henceforth a, can be uniquely determined by solving Dirichlet boundary 


value problems for (24) and (25). | 


Our next goal is to embed the abstract manifold (M, g) with conformally 
Euclidean metric into Q with metric (o(x))'6; ;. To achieve this goal, we use the 
a priori knowledge that such embedding exists and the fact that we have already 
constructed a corresponding to a(x) on M. 


Lemma 3. Let (M, g) be a compact Riemannian manifold, a(x) a positive smooth 


function on M, andy : dQ — 0M a diffeomorphism. Assume also that there is a 
diffeomorphism W : Q — M such that 


Vlg = Ve Wg = (a(Y(x))) by. 


Then, if 2, (M, g), a, and y are known, it is possible to construct the diffeomor- 
phism W by solving ordinary differential equations. 
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Proof. Let € = (z, tT) be the boundary normal coordinates on M\q. Our goal is to 
construct the coordinate representation for ¥~! = X, 


X:M\o>Q, 
R324) ke) 


Denote by hj; (x) = a(W(x)) 15; the metric tensor in Q. Let Ty j, = oe gipl ix 
be the Christoffel symbols of (2, 4;;) in the Euclidean coordinates and let ere be 
Christoffel symbols of (M, g), in ¢-coordinates. Next, we consider functions h; fs 
Ty. i;, etc., as functions on M \w in (z, t)-coordinates evaluated at the point x = 
x(z,T), for example, Ty j;(z,t) = xij (x (z, t)). Then, since W is an isometry, the 
transformation rule of Christoffel symbols with respect to the change of coordinates 
implies 


as dx! ax! ax* ” Ox! Ox! 
a ij hi; ; 27 
is » 7 Och lY ac ¥ x 2 age agHatY en 
where 
hj; (z,T) : 6 (28) 
if(%T) = ~——~ 0ij- 
Poe aW(z.1) ! 
Using Eqs. (27) and (28), we can write ae in the form 
0? x/ : gh e, OE pee 
pence ie ae _ JP = 
agape) = Dd, ates (Fae axe 243 age 
P.o,uv=1 n=1 
0c? 0c? ac? ax! ax” 
‘ E oa ae | age =): a 


As @ and Ci are known as a function of ¢, the right-hand side of (29) can be 
written in the form 


Ox/ ; ox 
ageage — Fite (« a 3 oe 


where F ao are known functions. Choose v = m, so that 


a2x/ _d ox/ 
acwace — dr \ ace)’ 


1224 M. Lassas et al. 


Then, Eq. (30) becomes a system of ordinary differential equations along normal 


geodesics for the matrix (0) . Moreover, since diffeomorphism WV: 
1 


Jb= ; 
dQ — dM is given, the boundary derivatives aes fe =1,...,n—1, are known for 
¢" = t = 0. By relation (28), 


ae” at an 


for €” = t = Owheren = (n!,...,n”) is the Euclidean unit exterior normal vector. 


Thus, a, 0) are also known. Solving a system of ordinary differential equations 


(30) with these initial conditions at t = 0, we can construct a, T) everywhere 


on M \a. In particular, taking « = n, we find a (z, t). Using again the fact that 
(x!(z,0),...,x"(z,0)) = w(z) are known, we obtain the functions x/ (z, t), z fixed, 
0 <tT < Tym (z), that is, reconstruct all normal geodesics on 2 with respect to metric 
hj;. Clearly, this gives us the embedding of (M, g) onto (Q, hij). |_| 


Combining the above results, we get the following result for the isotropic wave 
equation. 


Theorem 2. Let 2 C R"” to be a bounded, open set with smooth boundary and 
c(x) € C®(Q) be a strictly positive function. Assume that we know Q, c|ag, and 
the nonstationary Robin-to-Neumann map A oq. Then it is possible to determine the 
function c(x). 


We note that in Theorem 2, the boundary value c|aq of the wave speed c(x) can 
be determined using the finite velocity of wave propagation (9) and the knowledge 
of Q and Agq, but we will not consider this fact in this chapter. 


From Boundary Data to Inner Products of Waves 

Let u/ (x,t) denote the solutions of the hyperbolic equation (3), A?’ be the finite 
time Robin-to-Dirichlet map for Eq. (3) and let dS, denote the Riemannian volume 
form on the manifold (0M, gay). We start with the Blagovestchenskii identity. 


Lemma 4. Let f,h € CS°(0M x R+). Then 


/ ut (x, T)u"(x, T) dV,(x) = (31) 
M 


s/f (f(x, t)(A7TA)(x, 8) — (A7" f(x, A(x, 8)) dAS(x)dtds, 
2 Ji Jam 
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where 
L={(s,t): O<t+s<2T, t <s, t,s > O}. 
Proof. Let 
w(t,s) = i ul (x, thu! (x, 8) dV,(x). 
Then, by integration by parts, we see that 


(0? — 2) w(t, s) 


II 


a [aru! (x, t)ul(x,s) — ut (x, 1)d2u"(x,s)] dV, (x) = 


-{ [Au! (x, tu" (x, s) — uf (x,t) Au! (x, s)] dV (x) = 
M 
= =f Bau! u(s) ~ ul OB. guh(8)] dS) = 
0M 
= / [A2? ut (x, t)ul (x, s) — ut (x, t)A7 u(x, s)] dS,(x). 
0M 


Moreover, 
Wii=o = Wls=o = 9, 
I:wl=o = Isw|s—o = 9. 
Thus, w is the solution of the initial boundary value problem for the one-dimensional 
wave equation in the domain (t, 5) € [0,27] x [0,27] with known source and zero 
initial and boundary data (10). Solving this problem, we determine w(t, s) in the 
domain where ¢t + s < 27 andt < s (see Fig. 1). In particular, w(7, T) gives the 


assertion. |_| 


The other result is based on the following fundamental theorem by D. Tataru 
[77,79]. 


Theorem 3. Let u(x,t) solve the wave equation u,;, + Au = 0 in M x R and 
ulrx(o27,) = OvUlrx(0.27,) = 0, where § A T C AM is open. Then 


u=Oin Krn, (32) 
where 


Krr, = {(.t)€MxR:dgx,T) <M -|t-T)} 


1226 M. Lassas et al. 


Fig. 1 Domain of integration t 
in the Blagovestchenskii 
identity 


> 


(7, T) 


Vv 


2T s 
Fig. 2. Double cone of 
influence 
t=2T, 
t= T, 
t=0 


is the double cone of influence (see Fig. 2). 


(The proof of this theorem, in full generality, is in [77]. A simplified proof for 
the considered case is in [42].) 

The observability Theorem 3 gives rise to the following approximate controlla- 
bility: 
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Corollary 1. For any openT C 0M and T, > 0, 
clay {ul (-, Ti): f € CS°(E x (0, 7)))} = L?(M(F, T1)). 
Here, 
M(I,T,) = {x € M: d,(x,T) < Ti} = Kor, N{t = Ti} 


is the domain of influence of I’ at time 7, and L*(M(T,7\)) = {a € L?(M) : 
supp(a) C M(T, T;)}. 


Proof. Let us assume that a € L?(M(I, T;)) is orthogonal to all u/(-,7,), f € 
Co? (Lx (0, 7))). Denote by v the solution of the wave equation 


(7 + Av=0; vier, =0, inMxR, 
OVir=aT = 4; By yvlauxr = 0. 


Using integration by parts, we obtain for all f € CP° (r x (0, T)) 


T| 
i; / Flesyv(e,s) dS (x) ds =f alaul (x. Td Vy (2) = 0 
0 0M M 


due to the orthogonality of a and the solutions u/(t). Thus, virx0,7) = 0. 
Moreover, as v is odd with respect tot = 7}, that is, v(x, T; +s) = —v(x, T;—s), we 
see that v|rx(7,,27,) = 0. As u satisfies the wave equation, standard energy estimates 
yield that u € C(R; H'(M)), and hence ulawxe € C(R; H'/7(0M)). Combining 
the above, we see that vIrx(027) = 0, and as By »v|rxo27) = 0, we see using 
Theorem 3 that a = 0. a 


Recall that we denote u/ (t) = u/(-,t). 
Lemma 5. Let T > O andT; C 0M, j = 1,...,J, be nonempty, relatively 


compact open sets, 0 < T; < ‘es < T. Assume we are given (0M, gam) and the 
response operator A?" . This data determines the inner product 


Ti (fic fd = ful Ge tu! G0) dV) 


for givent > Oand fi, fo € Cf°(dM x R+), where 


J 


N=(\(M (Ty, Ti \\ M (T; .T;)) CM. 


j=l 
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Proof. Let us start with the case when f) = fp) = f and 7; = 0 forall j = 
| ee eee 

Let B = Ory x [T — T;,T]). For all h € Cj°(B), it holds by (9) that 
supp(u"(-,7)) C N, and thus 


lu! (T) ~~ ul (T) Ii2caeav,) 


=f (ule. T) we TY) dV) + f (ul (x,T))” dV, (x). 
N M\N 


Let 7 (x) be the characteristic function of the set N. By Corollary 1, there is h € 
Cp? (B) such that the norm l~wut (T) — ul (T)|z2(av,) is arbitrarily small. This 
shows that J," (fi, f2) can be found by 


INF A) = le Dr anaryy ~ ye inf FO (33) 
0 


where 


F(h) = |lu/ (T) — uD )lltaanav,): 


As F(h) can be computed with the given data (10) by Lemma 4, it follows that we 
can determine J) (f, f) for any f € C2°(0M x R4). Now, since 


In(h,. A) =-(Fi+ efit A)—- Ini ft. A- Ad), 


Ale 


the claim follows in the case when Le = Oforall 7 = 1,2,...,J. 
Let us consider the general case when 7; may be nonzero. We observe that we 


can write the characteristic function xy (x) of the set VN = ‘en (m (r jo Ti')\ 


j=l 
M (1.7) as 


Ki Ko 
awl) = Docexm— D2 eam Od, 
k=1 


k=Kj\+1 


where cx, € R are constants which can be determined by solving a simple linear 
system of equations and the sets Nx are of the form 


N= U M(T;,t;). 


JETk 


Wave Phenomena 1229 


where I, C {1,2,...,J} and t; € Pe TST aad U {Ty pe 
1,2,...,J}. Thus, 


Ki K2 
Inf A) = Docedh, A A- So dh, fi A) 
k=1 k=K,+1 


where all the terms J Mm (fi, #2) can be computed using the boundary data (10). 


From Inner Products of Waves to Boundary Distance Functions 
Let us consider open sets T; C 0M, j = 1,2,..., J and numbers es > T; > 0. 


For a collection {(t), T;, i) Df SH lyeews ih we define the number 
P (52a) j= ed) = sup Int A) 


where T = (max Tt) +1, 


J 


v= FV (m (ty.7¢)\M (0-77) 


j=l 


and the supremum is taken over functions f € C§°(dM x (0,T)) satisfying 
lu? (T)iln2cy < 1. When r Cc 0M, 7 = 1,2,...,J, are open sets, so that 
ry — {zj} as q — oo, that is, {z;} C i Cc i for all g and gry =1z;}, 
we denote 


P({(z).7}.77):f = 1---.F})= Jim P({(TE TF Ty) ey = --}). 


Theorem 4. Let {z,}°2, be a dense set on 0M and r(-) € C(0M) be an arbitrary 
continuous function. Then r € R(M) if and only if for all N > 0 it holds that 


P(f(ciren) + sre— a) i date wt) so, (34) 


Moreover, condition (34) can be verified using the boundary data (10). Hence, the 
boundary data determine uniquely the boundary distance representation R(M) of 
(M, g) and therefore determine the isometry type of (MV, g). 


Proof. “If”—part. Let x € M and denote for simplicity r(-) = r,(-). Consider a 
ball By;y (x) C M of radius 1/N and center x in (M, g). Then, for z € 0M 
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Byw(x) CM (are se x) \M (re - x) | 


By Corollary 1, for any T > r(z), there is f € Cf? (0M x (0, 7)) such that the 
function u/(-,7) does not vanish a.e. in Bij (x). Thus, for any N € Z4 and 
T = max{r(z,): n=1,2,...,N}, we have 


P (| (c.re)+ worn) x) a iL) 
> | lu (x, T)|? dV,(x) > 0 
Byjn (x) 


“Only if”—part. Let (34) be valid. Then for all N > 0, there are points 


N 
xy € Ay = () (™ (<r + x) \M (cre = x) (35) 


n=1 


as the set Ay has to have a nonzero measure. By choosing a suitable subsequence 
of xy (denoted also by xj), there exists a limit x = limy—+oo0 Xy. 
Let j € Z,. It follows from (35) that 


1 1 
r(zj)- WV < dg(xn,zj) <r(zj)+ W for all N > j. 


As the distance function d, on M is continuous, we see by taking limit N — oo 
that 


d,(x,zj) = r(z;), FS Deaton 


Since {z;}72 are dense in dM, we see that r(z) = dg(x, z) for all z € 0M, that is, 


r= he a 


Note that this proof provides an algorithm for construction of an isometric copy 
of (M, g) when the boundary data (10) are given. 


Alternative Reconstruction of Metric via Gaussian Beams 
Next we consider an alternative construction of the boundary distance representation 
R(M), developed in [6,41,42]. In the previous considerations, we used in Lemma 5 


the sets of type N = ‘em (u (Ty, T}')\ M (Ty, i) C M and studied 
the norms || yvu/ (-, T)|| 12(m)- In the alternative construction considered below, we 
need to consider only the sets N of the form N = M(To, 70). For this end, we 


consider solutions u/ (x,t) with special sources f which produce wave packets, 
called the Gaussian beams [3, 63]. For simplicity, we consider just the case when 
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A = —A, + q; 


and give a very short exposition on the construction of the Gaussian beam solutions. 
Details can be found in, for example, Ref. [42], Chapter 2.4, where the properties of 
Gaussian beams are discussed in detail. In this section, we consider complex valued 
solutions u/ (x, t). 

Gaussian beams, called also “quasiphotons,” are a special class of solutions 
of the wave equation depending on a parameter ¢ > O which propagate in a 
neighborhood of a geodesic y = yy <¢([0, L]), g(€,&) = 1. Below, we consider 
first the construction in the case when y is in the interior of M. 

To construct Gaussian beams, we start by considering an asymptotic sum, called 
formal Gaussian beam, 


N 
U,(x,t) = M, exp {—(ie)'0(x, t)} > u(x, t)(ie)*, (36) 
k=0 
where x € M,t € [t_,t4], and M, = (xe)7"/ 4 is the normalization constant. 


The function 6(x,f) is called the phase function and u,z(x,t), k = 0,1,...,N 
are the amplitude functions. A phase function 6(x, t) is associated with a geodesic 
th y(t) e Mif 


Im 0(y(t),t) = 0, (37) 

Im 0(x, t) > Cod, (x. y(0)’, (38) 

for t € [t_,t;]. These conditions guarantee that for any ¢, the absolute value 
of U,(x,t) looks like a Gaussian function in the x variable which is centered at 
y(t). Thus, the formal Gaussian beam can be considered to move in time along the 


geodesic y(t). The phase function can be constructed, so that it satisfies the eikonal 
equation 


(7.0 o) - ahs) : A(x, t) 2 4 j= 0 (39) 
apo BS ghd ag ale 


where = means the coincidence of the Taylor coefficients of both sides considered 
as functions of x at the points y(t), t € [t_, t+], that is, 


v(x,t) <0 if O&v(x,f)|x=y~) = 0 for alla € N” andt € [t_, ty]. 


The amplitude functions ux, k = 0,..., N can be constructed as solutions of the 
transport equations 


Loux = (07 —Ag+q) uk-1, with u-| = 0. (40) 


1232 M. Lassas et al. 


Here £y is the transport operator 
Lou = 20,00,u— 2(VO, Vu)g + (07 — Ag) O-u, (41) 


where Vu(x, t) = ae gl* (x) He (x, ny is the gradient on (M, g) and (V,W), = 
Va #/” (x) V; (x) W(x). The following existence result is proven, for example, 


in [3,42, 63]. 


Theorem 5. Let y € Me™ — € T,M be a unit vector andy = yyg(t), t € 
[t_, t+] C R be a geodesic lying in M™ whent € (t_, t+). 

Then there are functions 0(x,t) and u;,(x, t) satisfying (38)—(40) and a solution 
Uue(x,t) of equation 


(a? —-A, +4) u(x,t) =0, (x,t) € Mx [tL, t4], (42) 
such that 
|ue(x,t) — (x, 1)U-(x,1)| < Cye"™, (43) 


where N(N) — oo when N — oo. Here o € Cy°(M x R) is a smooth cut-off 
function satisfying ¢ = 1 near the trajectory {(y(t),t): t € [t-,t4]} CMxR. 


In the other words, for an arbitrary geodesic in the interior of M, there is a 
Gaussian beam that propagates along this geodesic. 

Next we consider a class of boundary sources in (3) which generate Gaussian 
beams. Let z € OM, f > 0, and let x b» z(x) = (z!(x),...,2"-'(x)) be a 
local system of coordinates on W C 0M near zo. For simplicity, we denote these 
coordinates as z = (z!,...,z’~!) and make computations without reference to the 
point x. Consider a class of functions f; = fe.29.19(z,f) on the boundary cylinder 
dM x R, where 


felz,t) = Buy (te) "4G, t) exp fie O@, D}V(z,1)). (44) 


Here ¢ € C§°(0M x R) is one near (zo, fo) and 


O(z,t) = —(t —t)) + 5 (Hole zo), (z—Z0)) + st =f)’, (45) 


where (-,-) is the complexified Euclidean inner product, (a,b) = }°a;b;, and 
Ho € C”*” is a symmetric matrix with a positive definite imaginary part, that is, 
(Ho) jk = (Ho)x; and Im Ho > 0, where (Im Ho) jx = Im (Ao) jx. Finally, V(z, £) 
is a smooth function supported in W x R+, having nonzero value at (zo, f9). The 
solution u/* (x,t) of the wave equation 
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du—Agu+qu=0, inMxR4, 
ul,-9 = 0rU|,29 = 0, (46) 


By tlaxe, = fe(z,t) 


is a Gaussian beam propagating along the normal geodesic y,,,,. Let S(Zo) € (0, 00] 
be the smallest values s > 0, so that y,,,(s) € 0M, that is, the first time when the 
geodesic y,,, hits to dM, or S(z) = oo if no such value s > 0 exists. Then the 
following result in valid (see, e.g., [42]). 


Lemma 6. For any function V € Cy°(W x Ry) being one near (z0, to), to > 9, 
and 0 < t) < S(z) and N € Z4, there are Cy so that the solution u(x,t) of 
problem (46) satisfies estimates 


|u/* (x,t) — P(x, t)U,(x, t)| < CyveX™, OK<t<tot+h (47) 


where U,(x,t) is of the form (36), for all 0 < ¢ < 1, where N(N) — oo when 
N > coand ¢ € Cf°(M x R) is o one near the trajectory (Vz (t),t + to): 1 € 
O.4}} MxR. 


Let us denote 


Py v(x) = XMvy,r) (x)v(x). 


Then, the boundary data (0M, gay) and the operator A uniquely determine the 
values Py .w? (Ol z2— for any f € CS°(0M x Ri), y € OM andt,t > 0. 
Let f; be of form (44) and (45) and u,(x,t) = u(x,t), f = f. be a Gaussian 
beam propagating along y-, ,, described in Lemma 6. The asymptotic expansion (36) 
of a Gaussian beam implies that for s < S(z) and t > 0, 


h(s), for dg(Yzv(s), ¥) < T, 
lim || Py.cte(. 5 + to) ll2a) = — (48) 
e>0 0, for dg(Yzv(S), ¥) > T, 


where f(s) is a strictly positive function. By varying t > 0, we can find 
dg (Veo,v(S). ¥) = rx(y), where x = y),(t). Moreover, we see that S(zo) can be 
determined using the boundary data and (48) by observing that S(zo) is the smallest 
number S > 0 such that if t; — S is an increasing sequence, then 


dg(Yin(Sk)-M) = ink, dy(Yoy0(8k).¥) > 0 as k — 00. 


Summarizing, for any z) € 0M, we can find S$(zo), and furthermore, for any 0 < 
t < S(Zo), we can find the boundary distance function r.(y) with x = y,),(t). As 
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any point x € M can be represented in this form, we see that the boundary distance 
representation R(M) can be constructed from the boundary data using the Gaussian 
beams. 


Travel Times and Scattering Relation 


We will show in this section that if (Q,g) is a simple Riemannian manifold, 
then by looking at the singularities of the response operator, we can determine 
the boundary distance function d,(x, y), x,y € dQ, that is, the travel times of 
geodesics going through the domain. The boundary distance function is a function 
of 2n —2 variables. Thus, the inverse problem of determining the Riemannian metric 
from the boundary distance function is formally determined in two dimensions and 
formally overdetermined in dimensions 1 > 3. 

Let Q C R"” be a bounded domain with smooth boundary. If the response 
operators for the two manifolds (Q,g,) and (Q, g2) are the same then we can 
assume, after a change of variables which is the identity at the boundary, the two 
metrics g; and go have the same Taylor series at the boundary [76]. Therefore, we 
can extend both metrics smoothly to be equal outside Q and Euclidean outside a 
ball of radius R. We denote the extensions to R” by g joJ = 1,2, as before. Let 
uj; (t,x, @) be the solution of the continuation problem 


Ou . 
Or — Agi uj => 0, in R” xR 
(49) 
uj(x,t) = d(t-—x-@),t <—R, 
where w € S""! = {x € R"; |x| = 1}. 


It was shown in [76] that if the response operators for (Q, g1) and (Q, g2) are 
equal, then the two solutions coincide outside Q, namely, 


u(t, x,@) = uw(t,x,@w), x €R”’\Q. (50) 


In the case that the manifold (Q,¢;), 7 = 1,2 is simple, we will use methods 
of geometrical optics to construct solutions of (49) to show that if the response 
operators of g; and go are the same, then the boundary distance functions of the 
metrics g; and g> coincide. 


Geometrical Optics 
Let g denote a smooth Riemannian metric which is Euclidean outside a ball of 
radius R. 

We will construct solutions to the continuation problem for the metric g (which 
is either g; or gz). We fix w. Let us assume that there is a solution to Eq. (49) of the 
form 
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u(x, t,@) = a(x, )5(t — o(x,@)) + v(x,0), u=0,t<—-R, (51) 


where a, are functions to be determined and v € L?,.. Notice that in order to 


satisfy the initial conditions in (49), we require that 


loc* 


a=1, $(x,@)=x-o forx-@ <—R. (52) 


By replacing Eq. (51) in Eq. (49), it follows that 


a ” 
ae — A,u = Ab" (t — (x, )) + BS! (t — o(x,0)) 
0? 
—(A,a)6(t — 6(x,@)) + aa Agv, (53) 
where 
n . ag To) 
_ _ ij pin 
A =a(x,o)| 1 Le at al (54) 
jk da * 
BR=2 ae wet ta 2. (55) 


jk=1 


We choose the functions ¢, a in the expansion (53) to eliminate the singularities 6” 
and 6’ and then construct v, so that 
d°v 
ay? —Agv= (Aga)d(t — p(x, w)), v=0,t<-—-R. (56) 


The Eikonal Equation 
In order to solve the equation A = 0, it is sufficient to solve the equation 


yg OOO: ag $(x,@) =x-w,x-w <—R. (57) 
oe ax! dxJ 


Equation (57) is known as the eikonal equation. Here we will describe a method, 
using symplectic geometry, to solve this equation. 

Let H,(x,&) = 4 oS gl (x)& Ej — 1) the Hamiltonian associated to the 
metric g. Note that the metric induced by g in the cotangent space T*R" is 
given by the principal symbol of the Laplace—Beltrami operator g—!(x, €) 
vi jai 8! (x) & §;. Equation (57) together with the initial condition can be rewritten 
as 

H,(x,d¢) =0, o(x,@) =x-0,x-w <—R, 
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where dp = )~;_, oe dx’ is the differential of ¢. 


Let S = {(x,&) : Hg(x,&) = 0}, and let My = {(x, Vo(x)) : x € R"}, then 
solving Eq. (57), is equivalent to finding @ such that 


Mg CS, with My = {(x,@);x-@ < —R}. (58) 


In order to find ¢ so that (58) is valid, we need to find a Lagrangian submanifold 
L, so that L C S,L = {(x,@);x-w < —R} and the projection of T*R" to R” is a 
diffeomorphism [32]. We will construct such a Lagrangian manifold by flowing out 
from N = {(x,@) :x-@ = s ands < —R} by the geodesic flow associated to the 
metric g. We recall the definition of geodesic flow. 

We define the Hamiltonian vector field associated to H, 


Vz = (SE -S). (59) 


The bicharacteristics are the integral curves of H, 


d Pcie Lym ag! 
tt mM] &. a iSj> = Ls cre (a 
as 2 gms): as 2 Xu axm ane ‘ on 


j=l 


The projections of the bicharacteristics in the x variable are the geodesics of the 
metric g and the parameter s denotes arc length. We denote the associated geodesic 
flow by 


Xg(s) = (xe(s), &e(s)) . 


If we impose the condition that the bicharacteristics are in S' initially, then they 
belong to S for all time, since the Hamiltonian vector field V, is tangent to S. The 
Hamiltonian vector field is transverse to N; then the resulting manifold obtained 
by flowing N along the integral curves of V, will be a Lagrangian manifold L 
contained in S. We shall write L = X,(N). 

Now the projection of N into the base space is a diffeomorphism, so that L = 
{(x, dy)} locally near a point of N. We can construct a global solution of (58) near 
Q if the manifold is simple. We recall the definition of simple domains. 


Definition 1. Let ( be a bounded domain of Euclidean space with smooth 
boundary and g a Riemannian metric on Q. We say that (Q, g) is simple if given 
two points on the boundary there is a unique minimizing geodesic joining the two 
points on the boundary and, moreover, dQ is geodesically convex. 


If (Q, g) is simple, then we extend the metric smoothly in a small neighborhood, 
so that the metric g is still simple. In this case we can solve the eikonal equation 
globally in a neighborhood of 2. 


Wave Phenomena 1237 


The Transport Equation 
The equation B = 0 is equivalent to solving the following equation: 


". ;, 96 da a 
Ds 87 Fel xt 1 Ze? =O Aek 


ij=l 


Equation (61) is called the transport equation. It is a vector field equation for 

a(x,q@), which is solved by integrating along the integral curves of the vector field 
_— yu ij 96 a : : 

v= 5218 . It is an easy computation to prove that v has length | and 


The solution of the transport equation (61) is then given by 


1 
a(x,@w) = exp (-; / ey) 5 (62) 
y 


where y is the unique geodesic such that y(0) = y, y(0) = w, y-w = O and y 
passes through x. If (Q, g) is a simple manifold, then a € C™(R"). 

To end the construction of the geometrical optics solutions, we observe that 
the function v(t,x,@) € Live by using standard regularity results for hyperbolic 
equations. 

Now we state the main result of this section in the following theorem. 


Theorem 6. Let (Q, gi),i = 1,2 be simple manifolds, and assume that the 
response operators for (Q, g1) and (Q, g2) are equal. Then dg, = dg,. 


Sketch of proof 1. Assume that we have two metrics g), g2 with the same response 
operator. Then by (50), the solutions of (49) are the same outside Q2. Therefore, the 
main singularity of the solutions in the geometrical optics expansion must be the 
same outside Q. Thus, we conclude that 


Pi(x,@) = d2(x,o), x ER"\Q. (63) 


Now $;(x, @) measures the geodesic distance to the hyperplane x -w = —R in 
the metric g. From this, we can easily conclude that the geodesic distance between 
two points in the boundary for the two metrics is the same, that is, dg,(x,y) = 
dy (x,y),xX,y € OQ. 

This type of argument was used in [61] to study a similar inverse problem for the 
more complicated system of elastodynamics. In particular, it is proven in [61] that 
from the response operator associated to the equations of isotropic elastodynamics, 
one can determine, under the assumption of simplicity of the metrics, the lengths of 
geodesics of the metrics defined by 


ds Stlejds, as Seixas, (64) 
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where d's. is the length element corresponding to the Euclidean metric and c,(x) = 


af ao Cs(X) = ie denote the speed of compressional waves and shear waves, 


respectively. Here A, jz are the Lamé parameters and p the density. 

Using Mukhometov’s result [56,57], we can recover both speeds from the 
response operator. This shows in particular that if we know the density, one can 
determine the Lamé parameters from the response operator. By using the transport 
equation of geometrical optics, similar to (61), and the results on the ray transform 
(see, e.g., [66]), Rachele shows that under certain a priori conditions one can also 
determine the density p [62]. 


Scattering Relation 

In the presence of caustics (i.e., the exponential map is not a diffeomorphism), the 
expansion (51) is not valid since we cannot solve the eikonal equation globally in 
Q. The solution of (50) is globally a Lagrangian distribution (see, e.g., [38]). These 
distributions can locally be written in the form 


u(t,x,@) = / el? x0. a(t, x, w,0) dé, (65) 


where ¢ is a phase function and a(t, x, w) is a classical symbol. 

Every Lagrangian distribution is determined (up to smoother terms) by a 
Lagrangian manifold and its symbol. The Lagrangian manifold associated to 
u(t,x,@) is the flow out from ¢ = x-@,t < —R by the Hamilton vector field 
of pelt, x,t,€) = 1? — ei gjk(x)&/E*. Here (zr, &) are the dual variables to 
(t,x), respectively. The projection in the (x, &) variables of the flow is given by the 
flow out from N by geodesic flow, that is, the Lagrangian submanifold L described 
above. 

The scattering relation (also called lens map) C, C (T*(R x 2)\0) x (T*(Rx 
9Q)\0) of a metric g = (g/) on Q with dual metric g~! = (g;;) is defined as 
follows. Consider bicharacteristic curves, y : [a,b] > T*(Q x R), of the Hamilton 
function pg(t, x,t, &), which satisfy the following: y(Ja, b[) lies in the interior, y 
intersects the boundary non-tangentially at y(a) and y(b), and time increases along 
y. Then the canonical projection from (Tig,49(R x Q)\0) x (TXaq(R x 2)\0) 
onto (7*(R x 9Q)\0) x T*(R x dQ)\0) maps the endpoint pair (y(b), y(a)) to a 
point in C,. In other words, C, gives the geodesic distance between points in the 
boundary and also the points of exit and direction of exit of the geodesic if we know 
the point of entrance and direction of entrance. 

It is well known that C, is a homogeneous canonical relation on ((T*(R x 
dQ)\0) x (T*(R x AQ)\0). (See [35] for the concept of a scattering relation.) Cy 
is, in fact, a diffeomorphism between open subsets of T*(R x 0Q)\0. 

In analogy with Theorem 6, we have the following theorem. 


Theorem 7. Let g;,i = 1,2 be Riemannian metrics on Q such that the response 
operators for (Q, g1) and (Q, go) are equal. Then 
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Cy, = Cy. 


Sketch of proof 2. Since by (49), we know the solutions of (48) outside Q. There- 
fore, the associated Lagrangian manifolds to the Lagrangian distributions u; must 
be the same outside (2. By taking the projection of these Lagrangians onto the 
boundary, we get the desired claim. 

In the case that (Q, g) is simple, the scattering relation does not give any new 


information. In fact ((t1, X1,T, &1), (to, Xo, T, £)) € Cg if t) — fo = dg (x1, Xo) and 


= 7 e140) j = 0,1. In other words d, is the generating function of the 


scattering relation. 

This result was generalized in [36] to the case of the equations of elastodynamics 
with residual stress. It is shown that knowing the response operator, we can recover 
the scattering relations associated to P and S waves. For this, one uses Lagrangian 
distributions with appropriate polarization. 

The scattering relation contains all travel time data; not just information about 
minimizing geodesics as is the case of the boundary distance function. The natural 
conjecture is that on a nontrapping manifold, this is enough to determine the metric 
up to isometry. We refer to [72] and the references therein for results on this 
problem. 


Curvelets and Wave Equations 


In this section we will discuss in more detail the use of curvelets in wave imaging. 
We begin by explaining the curvelet decomposition of functions, using the standard 
second dyadic decomposition of phase space. The curvelets provide tight frames of 
L?(R") and give efficient representations of sharp wave fronts. We then discuss 
why curvelets are useful for solving the wave equation. This is best illustrated 
in terms of the half-wave equation (a first-order hyperbolic equation), where a 
good approximation to the solution is obtained by decomposing the initial data 
in curvelets and then by translating each curvelet along the Hamilton flow for the 
equation. Then we explain how one deals with wave speeds of limited smoothness, 
and how one can convert the approximate solution operator into an exact one by 
doing a Volterra iteration. 

The treatment below follows the original approach of Smith [67] and focuses on 
explaining the theoretical aspects of curvelet methods for solving wave equations. 
We refer to the works mentioned in the introduction for applications and more 
practical considerations. 


Curvelet Decomposition 

We will explain the curvelet decomposition in its most standard form, as given 
in [67]. In a nutshell, curvelets are functions which are frequency localized in 
certain frequency shells and certain directions, according to the second dyadic 
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oN 2 . 


Fig. 3 A curvelet g, with y = (k,@, x) is concentrated (a) in the frequency domain near a box 


of length ~2* and width ~2*/? and (b) in the spatial side near a box of length ~2—* and width 
~27k/2 


decomposition and parabolic scaling. On the spatial side, curvelets are concentrated 
near lattice points which correspond to the frequency localization. 

To make this more precise, we recall the dyadic decomposition of the frequency 
space {£ € R"} into the ball {|&| < 1} and dyadic shells {2* < |&| < 2**'}. 
The second dyadic decomposition further subdivides each frequency shell {2* < 
|€| < 2*+1) into slightly overlapping “boxes” of width roughly 2*/? (thus each 
box resembles a rectangle whose major axis has length ~2* and all other axes have 
length ~2k/2), See Fig. 3a for an illustration. The convention that the width (2k/2) 
of the boxes is the square root of the length (2) is called parabolic scaling; this 
scaling is crucial for the wave equation as will be explained later. 

In the end, the second dyadic decomposition amounts to having a collection of 
nonnegative functions ho, h? € C9°(IR"), which form a partition of unity in the 
sense that 


1 = ho(&) + D> She’. 


k=0 @ 


Here, for each k, w runs over roughly 2—)*/? unit vectors uniformly distributed 
over the unit sphere, and h’?? is supported in the set 


gv cig sate, [Ea] <2r¥ 


g| 


We also require a technical estimate for the derivatives 
[(co, Dg)4 OEHe E)| < Chg 2 kU lel, 


with Cj. independent of k and w. Such a partition of unity is not hard to construct; 
we refer to [73, Sect. 20.9.4] for the details. 
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On the frequency side, a curvelet at frequency level 2" with direction w will 
be supported in a rectangle with side length ~2* in direction w and side lengths 
~2*/2 in the orthogonal directions. By the uncertainty principle, on the spatial 
side, one expects a curvelet to be concentrated in a rectangle with side length 
~2-* in direction w and ~27*/2 in other directions. Motivated by this, we define 
a rectangular lattice S? in R”, which has spacing 2-* in direction w and spacing 
2-*/2 in the orthogonal directions, thus 


n 
ER= 4xeR";x= a2 et > by2 "0; where a,b; € Z 
j=2 


and {@, @2,..., @y} is a fixed orthonormal basis of R”. See Fig. 3b. 
We are now ready to give a definition of the curvelet frame. 


Definition 2. For a triplet y = (k, w, x) with w as described above and for x € E®, 
we define the corresponding fine-scale curvelet p, in terms of its Fourier transform 
by 


dy (é) = Og Pe . 5h? (é). 
The coarse-scale curvelets for y = (0, x) with x € Z” are given by 


Gy (§) = (20) /7e"* Fao (E). 


The distinction between coarse- and fine-scale curvelets is analogous to the 
case of wavelets. The coarse-scale curvelets are used to represent data at low 
frequencies {|&| <1}, and they are direction independent, whereas the fine-scale 
curvelets depend on the direction w. 


The next list collects some properties of the (fine-scale) curvelets gy. 


¢ Frequency localization. The Fourier transform ¢@,(&) is supported in the shell 
{2k-1/2 < |&| < 2*+3/2) and in a rectangle with side length ~2* in the w 
direction and side length ~2*/? in directions orthogonal to w. 

* Spatial localization. The function g,(y) is concentrated in (i.e., decays away 
from) a rectangle centered at x € G?, having side length 2-* in the w direction 
and side lengths 2~*/? in directions orthogonal to w. 

* Tight frame. Any function f ¢ L*(IR”) may be written in terms of curvelets as 


£0) = do ¢%0), 
Y 


where c, are the curvelet coefficients of f: 
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Cy = im FY) Py (y) dy. 


One has the Plancherel identity 
[ roa = Die. 
y 


The last statement about how to represent a function f € L?*(R") in terms of 
curvelets can be proved by writing 


f (8) = hE SE) + VME LO 


k=0 @ 


and then by expanding the functions h?(&) f (€) in Fourier series in suitable 
rectangles, and finally by taking the inverse Fourier transform. Note that any L? 
function can be represented as a superposition of curvelets ~,, but that the gy, are 
not orthogonal and the representation is not unique. 


Curvelets and Wave Equations 
Next we explain, in a purely formal way, how curvelets can be used to solve the 
Cauchy problem for the wave equation 


(0? + A(x, Dx)) u(t, x) = F(t,x) inRxR’, 
u(0, x) = uo(x), 
0,u(O, x) = uy (x). 


Further details and references are given in the next section. Here A(x, Dy) = 
DS sie *(x) Dx, Dx, is a uniform elliptic operator, meaning that g/* = g'/ and 
0<A< ipa g/*(x)E;& < A < 00 uniformly over x € R" and & € S""!. We 
assume that g/* are smooth and have uniformly bounded derivatives of all orders. 

It is enough to construct an operator S(t) : uw; +> u(t, - ) such that u(t, x) = 
(S(t)u1)(x) solves the above wave equation with F = 0 and uo = 0. Then, by 
Duhamel’s principle, the general solution of the above equation will be 


u(t, x) = i. S(t —s)F(s,x) ds + (0,;S(t)uo) (x) + GS(t)u,) (x). 


To construct S(t), we begin by factoring the wave operator 0? + A(x, D,) into two 
first-order hyperbolic operators, known as half-wave operators. Let P(x, Dy) = 
J A(x, D,.) be a formal square root of the elliptic operator A(x, D,.). Then we have 
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a + A(x, Dy) = (8; —iP)(; + iP) 


and the Cauchy problem for the wave equation with data F = 0, uy) = 0, u,; = f is 
reduced to solving the two first-order equations 


(0, -iP)v=0, vO)=f 
(0, +iP)u=v, wu(0O)=0. 
If one can solve the first equation, then solvability of the second equation will follow 


from Duhamel’s principle (the sign in front of P is immaterial). 
Therefore, we only need to solve 


(0, —iP)v(t, x) = 0, 
v(O,x) = f(x). 


For the moment, let us simplify even further and assume that A(x, Dx) is the 
Laplacian —A, so that P will be the operator given by 


Pf (é) = (elf. 


Taking the spatial Fourier transform of the equation for v and solving the resulting 
ordinary differential equation give the full solution 


wy) = ny felted Fg) as. 


Thus, the solution is given by a Fourier integral operator acting on /: 


way) = Ony fei Pa(ty.8) FE) dB. 


In this particular case, the phase function is ®(¢, y,&) = y-&+t]&|, and the symbol 
isa(t, y,&) = 1. 

So far we have not used any special properties of £. Here comes the key point. 
If f is a curvelet, then the phase function is well approximated on supp(f ) by its 
linearization in &: 


Ot, y,€) + VeP(t, y,@)-E foré € supp(/). 


(This statement may seem somewhat mysterious, but it really is one reason why 
curvelets are useful for wave imaging. A slightly more precise statement is as 
follows: if W(t, y,&) is smooth for & 4 0, homogeneous of order 1 in &, and its 
derivatives are uniformly bounded over t € [—T, T] and y € R” andé € S"~!, then 
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whenever &-w ~ 2 and |£ — (E-w)a| < 2*/?. Also the derivatives of W(t, y, &) — 
VeW(t, y,@)-& satisfy suitable symbol bounds. Parabolic scaling is crucial here; 
we refer to [18, section “Travel Times and Scattering Relation’’] for more on this 
point.) Thus, if # = g, then the solution v with this initial data is approximately 
given by 


eS any" f elt) EB (E) dE = 9, (y + to). 


n 


Thus the half-wave equation for P = /—A, whose initial data is a curvelet in 
direction , is approximately solved by translating the curvelet along a straight line 
in direction w. 

We now return to the general case, where A(x, &) is a general elliptic symbol 


pe g*(x)&; &,. We define 


P(x, &) = VA, &). 


Then p is homogeneous of order | in , and it generates a Hamilton flow (x (t),& (t)) 
in the phase space T*R” = R"” x R", determined by the ordinary differential 
equations 


x(t) = Vep(x(t), €()). 
E(t) = —Vx p(x(), €(0). 


If A(x,&) is smooth, then the curves (x (t),&(t)) starting at some point 
(x (0), £(0)) = (x,w) are smooth and exist for all time. Note that if p(x, &) = |&|, 
then one has straight lines (x(t), &(t)) = (x + to, @). 

Similarly as above, the half-wave equation 


(a, —iP)v(t, x) = 0, 
v(0,x) = f(x) 


can be approximately solved as follows: 


1. Write the initial data f in terms of curvelets as f(y) = hae Cy Py(y). 

2. For a curvelet g,(y) centered at x pointing in direction w, let g,(t, y) be 
another curvelet centered at x(f) pointing in direction &(t). That is, translate 
each curvelet ~, for time ¢ along the Hamilton flow for P. 

3. Let v(t, y) = ay Cy@y(t, y) be the approximate solution. 
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Thus, the wave equation can be approximately solved by decomposing the initial 
data into curvelets and then by translating each curvelet along the Hamilton flow. 


Low-Regularity Wave Speeds and Volterra Iteration 

Here we give some further details related to the formal discussion in the previous 
section, following the arguments in [67]. The precise assumption on the coefficients 
will be 


glk (x) EC 1,1 (R"). 
This means that 0%g/* € L®(IR”) for |a| < 2, which is a minimal assumption 
which guarantees a well-defined Hamilton flow. 
As discussed in section “Curvelets and Wave Equations,” by Duhamel’s formula, 


it is sufficient to consider the Cauchy problem 


(8? + A(x, Dy))u(t,x) =0 inRxR", 


u(0O, x) = 0, 
0,u(0, x) = ff: 
Here, A(x, Dy) = Tpai f(x) Dy, Dy, and gi € C!(R"), g/* = g*/, and 


OA Sy ei gl*(x)E;& < A < co uniformly over x € R" andé € S$”. 
To deal with the nonsmooth coefficients, we introduce the smooth approxima- 
tions 


An(x.€) = > gf EE. gf = xO"? Dg" 


ij=l 


where y € C°°(R") satisfies 0 < y < 1, x(€) = 1 for |&| < 1/2, and 
x(€) = 0 for |&| => 1. We have written (x(2~*/? Dy )g) (€) = y(2-*/7£) 8 (E). Thus 
Cs are smooth truncations of g” to frequencies < 2*/*. We will use the smooth 
approximation A, in the construction of the solution operator at frequency level 2", 
which is in keeping with paradifferential calculus. 

Given a curvelet g,(y) where y = (k,@,,X,), we wish to consider a curvelet 
y(t, y) which corresponds to a translation of ~, for time ¢ along the Hamilton flow 
for Hy (x, &) = Ax (x, §). In fact, we shall define 


Py(t, VY) = by (O,() (y = xXy(t)) + Xy) , 
where x,,(t) and the n x n matrix ©,(f) arise as the solution of the equations 


x= Ve A(x, ), 
o= —V, Ak (x, @) - (a: Vx Ak (x, W))@, 
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Fig. 4 The translation of a curvelet ~, for time ¢ along the Hamilton flow 


0 = -—O(@ ® V; Ag (x, @) — Vy Ak (x, @) ®@ @) 


with initial condition (x, (0), @, (0), ©,(0)) = (x), @,, 1). Here v ® w is the matrix 
with (v @ w)x = (w-x)v. The idea is that (x) (t), wy (t)) is the Hamilton flow for 
Hy restricted to the unit cosphere bundle S*R"” = {(x,&) € T*R"; |&| = 1}, 
and ©,(¢) is a matrix which tracks the rotation of w, along the flow and satisfies 
©, (t)@,(t) = w, for all t. See Fig. 4 for an illustration. 

We define an approximate solution operator at frequency level 2° by 


ExOf0) = YS (Lorene). 


yhikl =k 


Summing over all frequencies, we consider the operator 


EOS =D EOS 


k=0 


This operator essentially takes a function f, decomposes it into curvelets, and then 
translates each curvelet at frequency level 2" for time ¢ along the Hamilton flow for 
A. 

It is proved in [67, Theorem 3.2] that E(t) is an operator of order 0, mapping 
A(R") to H%(R") for any a. The fact that E(t) is an approximate solution operator 
is encoded in the result that the wave operator applied to E(t), 


T(t) = (0; + A(x, Dx)) E(t), 


which is a priori a second-order operator, is in fact an operator of order 1 and maps 
Ht! (R") to H*(R") for —1 < a < 2. This is proved in [67, Theorem 4.5] and 
is due to the two facts. The first one is that when A is replaced by the smooth 
approximation A;, the corresponding operator 
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Yo (8 + An, Dx) Ex) 


k 


is of order 1 because the second-order terms cancel. Here, one uses that translation 
along the Hamilton flow approximately solves the wave equation. The second fact 
is that the part involving the nonsmooth coefficients 


¥5 (Ae, Dx) — A(x, Dx)) Ex) 
k 


is also of order 1 using that A, is truncated to frequencies < 2‘/? and using estimates 
for A — Ax obtained from the C'! regularity of the coefficients. 

To obtain the full parametrix, one needs to consider the Hamilton flows both for 
JA, and —./Ax, corresponding to the two half-wave equations appearing in the 
factorization of the wave operator, and one also needs to introduce corrections to 
ensure that the initial values of the approximate solution are the given functions. 
For simplicity, we will not consider these details here and only refer to [67, Sect. 4]. 
The outcome of this argument is an operator s(t, 5s), which is strongly continuous in 
t and s as a bounded operator H%(R") > H**!(R") satisfies s(t, s) f|;; = 0 and 
0,s(t, 5) f|r=s = f, and further the operator 


T(t,s) = (07 + A(x, Dx)) s(t, 8) 
is bounded H°(R”) > H°(R") for—1 <a <2. 
We conclude this discussion by explaining the Volterra iteration scheme, which 


is used for converting the approximate solution operator to an exact one, as in [67, 
Theorem 4.6]. We look for a solution in the form 


u(t) = s(t,0) f + if s(t, s)G(s) ds 


for some G € L! ([-to, fo]; H%(R")). From the properties of s(t, s), we see that u 
satisfies 


t 
(a7 + A(x, Dy))u = T(t,0) f + Gd) +f T(t, s)G(s) ds. 
0 
Thus, u is a solution if G is such that 
t 
G(t) + i T(t, s)G(s) ds = —T(t, 0) f. 
0 


Since T(t, s) is bounded on H*%(R”) for —1 < @ < 2, with norm bounded by 
a uniform constant when |t], |s| < fo, the last Volterra equation can be solved by 
iteration. This yields the required solution u. 
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4 Conclusion 


In this chapter, inverse problems for the wave equation were considered with 
different types of data. All considered data correspond to measurements made 
on the boundary of a body in which the wave speed is unknown and possibly 
anisotropic. The case of the complete data, that is, with measurements of amplitudes 
and phases of waves corresponding to all possible sources on the boundary, was 
considered using the boundary control method. We showed that the wave speed can 
be reconstructed from the boundary measurements up to a diffeomorphism of the 
domain. This corresponds to the determination of the wave speed in the local travel 
time coordinates. Next, the inverse problem with less data, the scattering relation, 
was considered. The scattering relation consists of the travel times and the exit 
directions of the wave fronts produced by the point sources located on the boundary 
of the body. Such data can be considered to be obtained by measuring the waves 
up to smooth errors or measuring only the singularities of the waves. The scattering 
relation is a generalization of the travel time data, that is, the travel times of the 
waves through the body. Finally, we considered the use of wavelets and curvelets 
in the analysis of the waves. Using the curvelet representation of the waves, the 
singularities of the waves can be efficiently analyzed. In particular, the curvelets 
are suitable for the simulation of the scattering relation, even when the wave speed 
is nonsmooth. Summarizing, in this chapter, modern approaches to study inverse 
problems for wave equations based on the control theory, the geometry, and the 
microlocal analysis were presented. 
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Abstract 

This paper deals with the inverse problem of the wave equation, which is 
of relevance in fields such as ultrasound tomography, seismic imaging, and 
nondestructive testing. We study the linearized problem by Fourier analysis, 
and we describe an iterative reconstruction method for the fully nonlinear 
problem in the time domain. We discuss practical problems such as the spectral 
incompleteness in reflection imaging and finding a good initial approximation. 
We demonstrate by numerical reconstructions from synthetic data what can be 
achieved. 


1 Introduction 


Imaging with ultrasound, or sonography, is a well-established technique in medicine 
and many other fields. It goes back to the middle of the last century [21]. The human 
body is scanned by a (usually handheld) transducer that emits the sound signals 
and records the reflected signals. The mathematical models used in present days 
are fairly simple. In particular they assume straight line propagation of the sound 
signals. More refined models take into account the bending of the rays. They go 
under the name of diffraction tomography [15, 32,56]. 

In this article we deal with an even more refined method of ultrasound imaging 
which is based on the wave equation. This permits to take into account not only 
bent rays but also multiple reflections. These techniques are not yet part of current 
medical practice, but prototype scanners based on these principles have already been 
built and tested in a clinical environment [4, 20]. 

Speaking in mathematical terms, we deal with the ultrasound imaging problem 
as the inverse problem for the wave equation. We determine parameters such as 
the speed of sound and the attenuation from measurements of the field outside 
the object. Imaging with the wave equation plays an important role not only in 
medical radiology, but also in nondestructive testing and seismic exploration. Our 
goal is to deal with the fully nonlinear problem. We start with a survey on the 
results obtained by linearization. For the fully nonlinear problem, we describe in 
detail the Kaczmarz method in the time domain. It turns out that Kaczmarz method, 
whose linear version is widely used in X-ray tomography [25], can be viewed in a 
very intuitive way as consecutive time reversal. We show by numerical examples 
that Kaczmarz method easily solves the standard problems, such as transmission 
tomography and reflection imaging with broadband data. We also study the behavior 
of the method in nonstandard situations, such as caustics, trapped rays, and, in 
particular, missing low frequencies in the source pulse. 

The literature on inverse problems for the wave equation is inexhaustible. So we 
restrict our discussion to the exact solution of the fully nonlinear coefficient inverse 
problem for general objects. We do not even mention the important work on obstacle 
scattering [12], where the object is homogeneous and only the shape of the object 
is sought for, nor do we deal with level set methods [18]. Neither do we discuss the 
many approximate methods, except for the Born approximation, because it gives so 
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much insight into the fully nonlinear problem. Other problems which we ignore are 
the linear inverse source problems [29] and phase contrast tomography [31]. 


2 The Model Problem 


We consider the following initial value problem for the wave equation. Let (2 be a 
domain in R",n = 2,3 and let ! = 092 . Let c(x) be the speed of sound in x € Q. 
Let T > 0 be the observation time. Let u be a solution of the initial-boundary value 
problem 


2 
a(x.) = 7 (x)Au(x,t), x €2,0<t<T, (1) 
ou 
ay q(t)drs(x), x ET, 0<t <T, (2) 
ue t) =O) t <0 (3) 


with v the exterior normal on J". We consider the inverse problem of determining c 
from the values of u(x,t) on J” x (0, 7), or on part of it. The function g represents the 
source pulse, and 57, is the Dirac 5-function on I” concentrated at the source s € I". 
We always assume that c?(x) = ca(x) /( + f(x)) with the (known) background 
velocity co and a function f > —1 which has to be determined. 

We remark that everything we are doing in this paper can also be done for more 
general problems, such as problems with varying density [26], in moving media [61] 
and problems with attenuation [4, 20, 40]. 


3 The Born Approximation 


Even though the underlying differential equation is linear, the inverse problem 
is nonlinear. Most of the literature on the problem makes use of some kind of 
linearization, mostly the Born approximation. Our goal is definitely to deal with 
the fully nonlinear problem. However linearization leads to valuable insights also 
for the nonlinear problem. Therefore we discuss the Born approximation in some 
detail. 


An Explicit Formula for the Slab 


We restrict ourselves to a very simple geometry: {2 is just the slab 0 < x, < D. 
Sources and receivers are sitting either on both boundaries (transmission mode) 
or on only one of them (reflection mode). It is more convenient to work with the 
inhomogeneous initial value problem 
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2 
ax t) = c*(x) (Au(x, +q@é(x—s)), xER",0<t<T, (4 


we ty=0, £20 (5) 


with 6 the Dirac function in R” and s a source. We also assume c(x) = co > O 
constant in this section. Equations (4), (5) is a simplified version of (1)—(3). 

We also look at the problem in the frequency domain. For the Fourier transform 
in R”, we use the notation 


f() = myn? I eT EF(xdx 


and we denote the inverse Fourier transform of f by f . Doing an inverse Fourier 
transform with respect to ¢ in (4), we get 


Aiu(x,w) + k7(1 + f(x, w) = —G(w)8(x — 5) (6) 


where kK = qw/co is the wave number. Because of (5) this equation has to be 
complemented by the Sommerfeld radiation condition for outgoing radiation, i.e., 


ou 


—— —ikiu > 0, |x| > c. 
d|x| 


In this section we derive an explicit representation of the Born approximation by 
Fourier analysis. We follow [14,42]. 

For f = 0, the solution of (6) is ¢(w)Gx(x — s) with G; the free space Green’s 
function which we give in the plane wave decomposition 


n_vrign GE! 
ss i(lxn le (E)—x" + &) 7 
Gx(x) = icy i= e KE) (7) 


where c. = 1/(4z), c3 = 1/(827), K(z) = Vk? — 22. For z > k, k(z) is defined 
to be i Vz? — k?. 
For arbitrary f, we put u(x) = G(w)Gx(x — 5) + v(x, 8), obtaining for v 
Axv(x,5) +71 + f(x) vx, 8) = —k?GGe (x — 8) f(x). 
Neglecting fv leads to the Born approximation 
Axv(x,8) + k?v(x, 8) © —k?G(w)Ge(x — 8) f(x) 
or 


v(x, 8) = K2G(w) [ Ge(x —y) Gey — 9) FO) dy. 


Sonic Imaging 1257 


With x’ the first n — 1 coordinates of x and correspondingly for s, y we rewrite this 
as 


D 
V(X", Xn 8", Sy) & qo) | [ 7 G(x! —y', Xn — In) f(s Yn) 
Gi(y’ 5 4 Yn — 5n)dy'dyn. 


Inserting (7) we get 


D 
ee ee ew -<ikq¢o) | i: / ei (Xn ynlkE)=C/— 9) 
’ ’ ’ n 
Rr-l J Rr-l 


i fs Ynetn- snlk(o’)—(y/—8') +0’) dé'do' 


—-——_dy ‘dyn. 
Keo) 


The y’ integral is just a (n — 1)-dimensional Fourier transform, hence 


V(X", Xn58",5n) &% —(270)"1k202G(@) fo? font fpnt 


ei ln rule’) Yn sale(o’) ee ea) F(a! ~~ Es Yn) cee dyn. (8) 


where f is the (n — 1)-dimensional Fourier transform of f with respect to the first 
n — | variables. Fourier transforms of dimensions n — 1 with respect to x’ and 5s’ 
yield 


(22)2"—) 
K(E)K(0") Jo 


D 
A ~ ix — 7 io — , 
w(E", Xn a", Sn) ~ —k?c?G(w) eilkn ynlk(EV+IYn—Sn |KO") 


fe + 0, Yn)dYn. 


In particular we have 


5(€’,0,0',0) ~ Af (E’ +0’, —K(é’) — K(o")), (9) 
0(&', D,0',0) = Ae?) fF (& + 0, K(E) — K(0')) (10) 


where ri is now the n-dimensional Fourier transform of f and A = 
—k?G(w)c2 (21)"/?" /(k (EK (o")). Equations (9), (10) is the solution of the 
inverse problem of reflection and transmission imaging, resp., in the Born 
approximation. Note that the derivation of (9), (10) depends entirely on the plane 
wave decomposition of G; which was used first in this context in [14]. 

Let’s look at the transmission case, i.e., (10). For simplicity, we consider the 2D 
case. Consider the semicircle of radius k in the upper half plane around the origin. 
Attach to each point of this semicircle the semicircle of radius k around this point 
that opens downward. According to (10), f is determined by the data along each 


1258 F. Natterer 


En 


Fig. 1 Left: Fourier coverage for transmission imaging. Right: Fourier coverage for reflection 
imaging. k is the wave number. The figures are for n = 2. For n = 3, one has to rotate the 
figures around the vertical axis 


of these semicircles. These semicircles fill the circles of radius k with midpoints 
(+k,0); see Fig. 1. In the reflection case (9) we proceed in the same way, except 
that we start out from the semicircle in the lower half plane. Now the attached 
semicircles fill the semicircle of radius 2k in the lower half plane, except for the 
circles of radius k around (+k, 0). 

Thus we see a fundamental difference between transmission and reflection 
imaging: In transmission imaging low frequency features f can be recovered 
irrespectively of the frequency content of the source pulse g, whereas in reflection 
imaging one needs low frequencies in the source pulse to recover low frequency 
features in f. This is one of the main difficulties in seismic imaging; see [9, 19, 22, 
30,60]. To the best of our knowledge, Fig. 1 appeared first in [35,64]. The difficulties 
in reflection imaging without low frequencies will be discussed in section “Missing 
Low Frequencies”. 

We also conclude from_ (9), (10) that combined transmission and reflection 
measurements determine f within the ball of radius 2k. In other words, since 
A = 2z/k is the wavelength of the irradiating sources, the spatial resolution 
according to Shannon’s sampling theorem is 27/2k = 4/2. This is a classical 
result already known to Born. 

The numerical evaluation of (9), (10) is by no means trivial. In principle it 
can be done by Fourier transforms. However this requires a sophisticated software 
for non-equispaced fast Fourier transforms. Assuming uniform sampling of > in 
&’ and o’ leads to non-equispaced sampling of f on a grid of the type shown in 
Fig. 2. An implementation similar to the filtered backprojection algorithm of X-ray 
computerized tomography has been given in [14]. 


An Error Estimate for the Born Approximation 


We now assume that {2 is a ball of radius r, the background speed is constant as in 
the previous section, and that the illumination done is by plane waves. This leads to 
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Fig. 2. Non-equidistant grid Tittiitie e 
at which f is given by (9), gsietes 6% 5 eeeses 
(10) forn = 2 . ee = 


the problem 
Au+k?(14+ f)u=0, (11) 
u= uj + Us (12) 
where u; = expikx-@ is the incoming wave with direction 9 € S"~! and 


wave number k and u, is the scattered wave satisfying the Sommerfeld radiation 
condition of exterior radiation. We assume f to be supported in 2. 6 runs 
through all of S' for n = 2 and through a great circle of S* for n = 3. The 
problem is to determine f from gg(z) = us(s@ + z) for some s with |s| > r 
and z € 6+. The decisive tool for the error estimate is an estimate for the 
operator 


Gyeu(x) = k? / Gi (x — y)u(y)dy 
in Lo(|x| <r). In [41] it has been shown that 


Gil |nodxi<r) = kr yn (kr) 


where y, is a function with 7, = SUP(o,o0) Yn < 00. For instance, yo < 0.8. We 


have us = k?Gx(f(u; + us)), hence, with g = kryn(kr), a\|fllzcaslen < 
1 


Al lnodxl<r): (13) 


q 
lus ||zo(xi<r) S 
ae 1—@||f \lzoodxl<r 


We introduce the propagation operator 


(Us fy = / Gx(s0 +2 y) Fyui(y)ay: 
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see [14]. For the (n — 1)-dimensional Fourier transform of Ug in 6+, we 
have 


Arey = 7. | oils 1 F _ 
(Us f) @)=i/%e eo ee k)@ + &) 


II 


where K(¢) = Vk? —|€|? and € 
[50]. 
We have 


sgn(s). This is essentially Theorem 3.1 of 


k?¢9 =Uo f + Us f 
hence 
k7¢ _ > au ils|«(C) 1 - A 
Bole) =i yf Fee (Flew) —1)9 +) + Us FY (Cen(0)8 +09). 


Now let € € R” be in the ball |E| < 2k. We can find 6 € S"~!, ¢ € 6+ with 
— = (ex(¢) —k)6 + C¢. For any such choice of 6 and ¢, the last formula yields 


FG) = fa) — us fYE + k9), falG) = -i y[Zerm(Ek*ao(0). 
Since 


[us fY*| < Qx)"? |lus|[zoaxten lS lzedeler) 
we have for |E| < 2k 


q 
1=4||f llzectxter) 


f(§) — fal@)| < Qn)" Fle ianeae (14) 


Thus we obtained an approximation of second order in f for a in the ball |&| < 
2k. This is what we expect from the Born approximation. The restriction to the 
bandwidth 2k is also natural as this corresponds to the maximal spatial resolution 
of 2/k. 

We see from (14) that the decisive quantity for the Born approximation to be 
valid is ¢|| f'||z,.(\x|<r)- In the engineering literature, see, e.g., [32], it is well known 
that this is a measure for the phase shift generated by the object f. Thus we arrive 
at the conclusion that the Born approximation works if the phase shift is sufficiently 
small. We also conclude from (14) that the reconstruction process, if restricted to the 
resolution z/k is fairly stable: there are no unstable operations in the computation 
of fr. It has been shown in [53] that this is also the case for the fully nonlinear 
problem, provided the geodesics behave reasonably. Thus the frequently discussed 
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instability of the inverse scattering problem is just a consequence of an inadequate 
choice of the frequency of the irradiating waves. 


4 The Nonlinear Problem in the Time Domain 


In this section we deal with the problem in the form (1)-(3). We suggest a very 
simple but practically useful iterative method and discuss some theoretical and 
practical aspects of it. 


The Kaczmarz Method in the Time Domain 


For a finite number of sources s;, 7 = 1,..., p we have to solve a system of the 
form 


RP=te7 tas (15) 


where R; is an operator between a Hilbert space H of functions on 92 x (0, T) and 
certain Hilbert spaces H; of functions on J” x (0,7). In our case the operator R; 
is defined by R;(f) = u; Irx(,T)? where u; is the solution of (1)—(3) for the source 
S= Sj. 

A natural method for solving (15) is the Kaczmarz method. This is an iterative 
method with the update 


fiat = Fp HRA) Cr (ey — Re), 7§ =1,2.-.. 


where j’ = j mod p . Here, R’; is the Fréchet derivative of Rj, Ri” its adjoint, 
and C; is a positive definite operator. Going through all the equations once is called 
a complete sweep of the Kaczmarz method. 

In the linear case, ie., if R; : H — Hj is a linear bounded operator the 
convergence of the Kaczmarz method is well understood, in particular for the case 
of CT; see, e.g., [50], Theorem 5.1. 

If Cj — R; Rj is positive semidefinite, the sequence f; converges for 0 < 
a < 2 to Pf, + R*g, where P is the orthogonal projection on the nullspace of 
R= (k,,...,R ) andR* is the Moore—Penrose generalized inverse of R. In the 
nonlinear case the situation is much less clear, see [6] for a theoretical discussion of 
the issue. 

The Kaczmarz method is not to be confused with the Landweber iteration. In 
the Landweber method one does the update only after all the equations have been 
processed, while in Kaczmarz we do the update as soon as a single equation is 
processed. In practice Kaczmarz turns out to be much faster than Landweber. In 
the linear finite dimensional case Kaczmarz is identical to the SOR method, see 
Theorem 5.2 in [50]. Also, the speed of convergence depends on the ordering 
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of the sources. Sequential ordering does not yield the best results. Sophisticated 
arrangements, in particular random orderings, are much better. For the case of CT, 
see Section 5.3.1 of [50]. 

In order to compute Ri (f) for our imaging problem, we replace f by f +h 
and u; by u; + w with h, w small and ignore higher order terms. We get for w the 
differential equation 


1 ow h uj 

gop 2 oe xE2,0<t <T, (16) 
dw 
rae xer, 0<t <T, (17) 
v 
w=0, t <0. (18) 


Hence we expect that 
Ri (f)h = wirx,7)- 


In fact it has been shown in [16] that with a suitable choice of Hilbert spaces H, H; 
the operator R; is Fréchet differentiable and the expression above is the derivative. 
A possible choice is H = H*(Q2), restricted to functions f > —1, and H; = 
H!'?(r x (0,T)). 

Now we come to the problem of computing the adjoint of R’,. For this we have 
to specify the spaces H, H;. To make things easy, we put H = L2(Q),H; = 
L2(I'x(0, T)), ie., we consider R’, as an unbounded operator between these spaces. 
Then the adjoint R’, satisfies 


(Ri (AA, g)ix12xo.ry = (h, RF) 8) Lrx00.7)) 


for sufficiently smooth functions h, g. Determining the exact domain of definition 
of the adjoint is a more tricky question which is not considered here. 

To find an explicit form of the adjoint, we follow [38]. We start out from the 
identity 


ae ea Tr) #z 
im (age 4w) eas ff (cosa 42) wea 
1 (ow Oz z 
=-[[ (j» = 25) ard +f = (Fe- we )ael 


where v is the exterior normal on I”. This holds for any sufficiently smooth functions 
w, zon $2 x(0, 7). Choosing for w the solution of (16)—(18) with u = u; the solution 
of (1)-(3) for source j and for z the solution of the final value problem 
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07z 


ae c?Azin 2 x (0,T), (19) 
Oz 
a = OM P07), (20) 
v 
z=Ofort>T (21) 


we obtain 


T 
"uj 
_ adtdx = ff R'.(f)h)dtdx, 
Lal = ff exycnin 


or, using inner products, 


1 7 ®u; 
—{h,— J at) = ( RP’ h, : 
( i, ar2 * ae ( if) on 


Hence 


1 pe Ou; 


(Ri) g=—a J, 


zdt (22) 


with z the solution of the final value problem (19)-(21). Note that z is nothing but 
the time reversed field. Thus the Kaczmarz method is just a consecutive form of 
time reversal, a technique that is used extensively for imaging problems [2, 3,57]. 


Numerical Example (Transmission) 


In this section we show what can be achieved for a mammography scanner patterned 
after [20] for the Salt Lake City breast phantom suggested in [4]. The reconstruction 
region is a circle of radius 8cm. On the boundary we have 128 receivers and a 
modest number of sources, namely 8, see Fig. 3. The source pulse at the sources 
is q(t) = cos(wt)exp(—(t/t)*) with t = m/w. The frequency of the irradiating 
waves is 1 MHz, i.e., w = 2710°/s. With a background speed co = 1,500 m/s this 
corresponds to a wavelength of 1.5mm, hence to a spatial resolution of 0.75 mm. 
In Fig. 4 we display the rays, suggesting that a straight line assumption is useless. 
We show the reconstruction after 1 sweep and after 6 sweeps of the Kaczmarz 
method with relaxation parameter a = 2 x 10! and C ; = 1. Note that the 
number of sources as well as the number of sweeps is very small, leading to very 
reasonable reconstruction times. The spatial stepsize was chosen to be 0.33 mm, 
which corresponds to 1/6 of the wavelength. 
For truly 3D examples, see [23, 39]. 
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Fig. 3 Kaczmarz method in time domain for breast phantom. Left: Phantom with 8 sources and 
128 receivers. Middle: One sweep of the Kaczmarz method. Right: Six sweeps of the Kaczmarz 
method. Grey window in all panels is from —0.1 to 0.1 


vidth 


: 
Bi lg 


Fig. 4 Rays for breast 
phantom 


/ nant 


Numerical Examples (Reflection) 


In this example the reconstruction region is the slab 0 < x, < D, and we have 
sources and receivers on x, = 0 only. This is the situation, e.g., in seismic imaging 
and in supine mammography [17, 26, 44]. In our example we adjust the Salt Lake 
City breast phantom of the previous section to the supine position of the patient, 
see Fig. 5. The five tumors have a diameter of 1.5mm. To discover them, we need 
incoming waves of wavelength 3mm, which, assuming a background speed of 
sound of 1,500 m/s, corresponds to a frequency of 500 kHz. From the section on the 
Born approximation we know that for reflection imaging we need low frequencies. 
So we are not surprised that the Kaczmarz method, as probably any other iterative 
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| | 


Fig. 5 Left: Breast phantom for patient in supine position. Right: Reconstruction 


method, does not work for the comparatively high frequency of 500kHz. What 
we do to circumvent this difficulty is to start the iteration with data for 30 kHz, 
use the result as an initial approximation for 1.2 MHz, and use the result again 
as an initial approximation for the final reconstruction with 500 kHz. This idea of 
starting with low frequencies is due to [8]. On each level we do only 6 sweeps, 
with relaxation parameters a chosen by trial and error. From Fig.5 we see that the 
tumors are easily recovered, as we expect from the theory. Height of the pictures 
is 7.5cm. The numerical work has been done on a 20cm x 10cm rectangle with a 
step size of 0.5 mm. We remark that only 19 sources sitting on the top boundary of 
the reconstruction region were used, but the number of receivers (400) was much 
larger. In the computation we did not use these 19 sources consecutively. We rather 
chose the sources in a random order. 


5 The Nonlinear Problem in the Frequency Domain 


This problem was considered in the early nineties as the paradigm of a nonlinear 
ill-posed problem. It was treated by regularized versions of Newton’s method. 
Numerical examples were restricted to objects of the size of the wavelength, see 
(24, 27, 33]. However in medical applications, e.g., in mammography, the object 
size is typically 100 wavelengths. 


Initial Value Techniques for the Helmholtz Equation 


In order to carry out the Kaczmarz method in the frequency domain, we have to 
solve the forward problem (6) and the corresponding adjoint problem repeatedly. 
In particular for high wave numbers k this is a numerical challenge [27]. For- 
tunately it is possible to use initial value techniques for the solution of elliptic 
equations such as (6). Such methods are notoriously unstable. However it is 
possible to stabilize them, simply by removing the evanescent waves, see [34, 
48, 51]. These methods are effective only if the true speed of sound deviates 
from the background only a little, typically by not more than 10 %, as is 
the case in mammography. An improved version that is able to handle much 
greater variations of the velocity, as they occur, e.g., in seismics, can be found 
in [59]. 
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The Kaczmarz Method in the Frequency Domain 


Based on the initial value techniques of the previous section, an extremely efficient 
implementation of the Kaczmarz method in the frequency domain is possible [49]. 
We describe the method for §2 the ball of radius r and plane wave irradiation. f 
may now be a complex-valued function 


ci i i 2y(x)co 


c(x)2 k c(x) 


f@) = 


with y(x) the (unknown) attenuation. Let "= be the planes or straight lines 


perpendicular to 6; touching I’. We assume the functions 8; = us on I to 


be known, ie., /’;~ play the role of the detectors. The operator Rj : L2(9%2) > 
LT), where u; is the solution of the initial value problems for (11) with 


Us = 87 
Oils 
OXy 


(x') = Qn) 0? [ - iK(E')us(Ee™ 8 dé! 


on I, where x’, & are the local variables on I7, K(&') = Vk? — |&’/? 
and ~ stands for the (n — 1)-dimensional Fourier transform with respect to x’. 
The last equation is a finite form of the Sommerfeld radiation condition, see, 
e.g., [11]. As in section (4.1), see also [38], we compute the adjoint of the 
derivative (Ri,(f))* : Lo(T;*) —> L(S2): (Ri (f)) * (r) = z, where z is the 
solution of 


Az+k?(1+ f)z = 0 between > and r+, (23) 
F) 
z=0, - = Fu; on Tt. (24) 
J 


We remark that (23), (24) is just the frequency domain form of time reversal. 


6 Initial Approximations 


The biggest problem with Kaczmarz method is to find an initial approximation for 
which the process converges. In this section we derive a heuristic criterion that 
works surprisingly well in practice, at least for transmission imaging. The condition 
reads 


| [r= firds| =a (25) 
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where A is the largest wavelength contained in the spectrum of the pulse g and the 
integral is taken along the geodesics belonging to the initial guess cp for c. 

We give two derivations of (25). Both of them are heuristic but fortunately they 
agree, and they perform well in practice. The first derivation uses the frequency 
domain formulation (6) of the problem. The forward operator R; is defined by 
R,(f) = alr with u the solution of (6) with s = s;. With w = ua — uo, uo the 
solution of (6) with f = fo, s = s; we have 


Rj(f) — Rj (fo) = wr, (26) 
—Aw- K+ fow= K(f — fait (27) 
For simplicity, we assume that we do the Kaczmarz method with C; = 


R’ ( Fi )(R; (f;))*. Then the first approximation /, of the Kaczmarz method 
satisfies 


Ri (fof — fo) = «(Rj (Sf) — Rj (fo). 
Let wy be the solution of 
— Aw, —-k7 (1+ fom =k’ (fi — fo)ito/ea. (28) 
Then 
wilr = Ri (fo)(A — fo)/o = Ri(f) — Rj (fo). 


Thus the solutions w, w, of (27), (28) coincide on J”. Comparing (27) and (28), we 
see that f; can be similar to f only if #, i have similar phase, 


|phase(u) — phase(uo)| < z, (29) 
say. If this condition is not satisfied, then even the first iterate f; goes astray. 
Thus we consider (25) as a highly necessary condition for the convergence of the 
Kaczmarz method with starting element fo. According to the WKB-approximation 
of u, we have 

ux Aexp(ik®) 
where the phase © satisfies the eikonal equation |V®|? = 1+ f and A is the real- 


valued amplitude. The same holds for % with an amplitude Ao and a phase ®o. For 
f close to fo, we have 


@ xb +5/(f-fds 
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Fig. 6 Example for 
non-convergence if (25) is not 
satisfied. Left: True object. 
Right: Object to which the 
method converges 


where the integral is along the geodesics of the background; see [53]. Combining 
the last three equations leads to (25). 

The second derivation of (25) takes place in the time domain. We look at the 
traces u(x’,0,t) and uo(x’,0,t). Qualitatively they look similar: They resemble 
time delayed and scaled versions of the pulse g. The delays are roughly the 
integrals of 1/c,1/co, resp., along the geodesics. Thus the difference of the delays 
is approximately 


[¢- [S- i (vi+F- ih) ~=f{5 Ae. 


The width w of the pulse g is determined by the lowest frequency w in the pulse. 
We have approximately w = z/w. Minimization of the norm of the difference of 
the traces has an effect only if the traces overlap, i.e., if the difference of the delays 
is smaller that the width of the pulse. This means that 


f-fods 
fo a 


Since A = 2c /a, this is equivalent to (25). 

We remark that (25) has been derived in [54] as the condition for the validity of 
the ray method for the inverse problem. 

In order to confirm (25) numerically we created a simple test phantom which is 
0.125 in the circle of radius 6 cm, the ambient speed of sound co being 1,500 m/s. It 
was illuminated by 8 sources sitting on the circle of radius 8cm. The sources had a 
constant signature from 350 down to 120 and 80kHz, respectively. In this case the 
left hand side of (25) is 1.5cm for fo = 0, while the largest wavelength is 1.25 and 
1.875 cm, respectively. Thus (25) is satisfied for the lowest frequency 80 kHz, and in 
fact we observed convergence to the true object. For the lowest frequency 120 kHz 
(25) is not satisfied. The method became stationary at an object that is quite different 
from the true object; see Fig. 6. We remark the highest frequency (350 kHz) is of no 
relevance here. If we double it, the convergence behavior is the same. 
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7 Pecularities 
Missing Low Frequencies 


Often the source pulse g contains only high frequencies, i.e., g (the signature in 
the jargon of seismic imaging) vanishes near 0. As we have seen previously this 
is not much of a problem in transmission imaging; it only forces us to use a good 
initial approximation. But it is one of the main difficulties in reflection imaging. 
We mentioned this in the framework of the Born approximation in Sect.3. Again 
we restrict ourselves to the two-dimensional case. Assuming that the source pulse 
q has wave numbers in [Amin, kmax], we conclude from (9) that f is determined by 
the reflection data inside the circle of radius 2k max, except for the circles with radius 
Kmin around (+Kmin, 0); see Fig. 7. This means that the reconstruction is essentially 
a high pass filtered version of the true object. 

For special media, we can do better [47]. For instance, if the medium in the slab 
0 <x, < D is layered, i.e., f depends on x, only, then the n-dimensional Fourier 
transform of f becomes fe. E,) = (2n)@-YD/26(€ FG) where 6 is the Dirac 
6 function in R’~! and f on the right hand side is the one-dimensional Fourier 
transform. In the reflection case we have x, = s, = 0 in (8), and since the problem 
is invariant with respect to translation in R”~! we can restrict s’ to 0. Hence (8) 
assumes the form 


v(x",0,0,0) & enka" 7G(w) fe FF (—2K(E') = P 
Rr-! 


or 


(Ay + k)v(x’,0, 0,0) & c2k?(22)""/?G(w) ee Fae Ede. 


Ria! 

7 (30) 

If x’ runs through all of R’~', we can recover f from this relation in the interval 
[0, 2k]. However in practice x’ is restricted to a finite aperture |x’| < R. This makes 


Fig. 7 Fourier coverage for 
reflection data with smallest 
wavenumber kynin and largest 
wavenumber kK max 
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Fig. 8 Left: Luneburg lens with rays focussing on a focal point on the rim. Middle: Luneburg lens 
with a few tumors. Right: Reconstruction with 6 sweeps of the Kaczmarz method 


it difficult to determine a in all of [0,2]. For instance, let f represent a point 
object in depth z, ie., f(xn) = 8(Xn — 2). Then, f (—2«(E")) = (2ar)“V/2e2*(8) ig 
a smooth function away from |&’| = k, but oscillates rapidly near this manifold, in 
particular for large z. In the vicinity of a point &’, |&’| < k it looks like a function of 
bandwidth 


—2z|Verk (E)| = 22]8'1/« (E'). 


Thus, by the sampling theorem, ‘a (&’) can be recovered from (30) only if R > 
2z|&’|/«(&'), ie., for |&’] < k/V1+422/R?. Summing up we arrive at the 
following conclusion: 

We can determine ri (&’) for objects up to depth z from reflection data with 
aperture R and wave number k only for 


k/J1 + 42/R? < |é"| < 2k. (31) 


The derivation for (31) given here is entirely heuristic, but it has been corroborated 
in many numerical experiments. For an alternative derivation, see [62]. Making use 
of analytic continuation one can go beyond this result and determine the remaining 
part of f provided that k, R, z satisfy a certain condition, see [47]. 

Another case in which we can do something without low frequencies is the case 
of media with a small dip angle, see [46]. 


Caustics and Trapped Rays 


It seems that the Kaczmarz method is very robust with respect to pecularities, such 
as caustics and trapped rays. As an example in which caustics occur we treat the 
famous Luneburg lens [36], which generates a focal spot on the rim of the lens, see 
Fig. 8. We see that the Kaczmarz method does not have the slightest problem with 
the reconstruction. 
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Fig. 9 Top left: Medium with trapped rays. Middle: Trapped rays medium with tumors. Right: 
Reconstruction with 6 sweeps of the Kaczmarz method 


With trapped rays we have the same situation. In Fig. 9 we display the ray pattern 
for a medium with trapped rays [28]. We add some tumors and do the reconstruction. 
As in the previous example the Kaczmarz method does the reconstruction without 
any trouble. 


The Role of Reflectors 


It belongs to the lore of seismic imaging that reflectors greatly improve the 
reconstructed images; see [10, 13, 63]. In this section we study this phenomenon 
within the Born approximation and we demonstrate its practical relevance by 
numerical simulations; see also [45]. 

We again assume {2 to be the strip 0 < x, < D. We use the form (4), (5) of 
the inverse problem. We put the sources on x, = D and model the reflector at 
Xn = 0 by stipulating the Neumann boundary condition du/dx, = 0 on x, = 0. 
Proceeding exactly as in section “An Explicit Formula for the Slab” we get for the 
Born approximation 


i Sis SEES [ G(x! — yin ynGR(" — 8", nr Se) FAY 


where 


G(x’ — vy! Xn Yn) = Gu(x! — y', Xn — Yn) + G(x! — y', Xn + Yn) 


is Green’s function for the Neumann boundary condition at x, = 0. We first do 
(n — 1)-dimensional Fourier transforms with respect to x’ and s’, obtaining 


D 
bE’, Xn. a’, Sn) x k?G(w) / / GO(é", Xn. yn)G(o', Yn, sale iE +9)" dy! dyn, 
QO Rr-l 
(32) 
where the hat stands for the (7 — 1)-dimensional Fourier transform. From (7) we see 
that 
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Gi’, xn) = 12)? aes 
k Xn) = U2 Ch F 
K(§') 


Thus, 


é G = ef en —YnlKED + eiltntynlk Ee) 
GRE ns Yn) = EQ)", 


k(&’) 
It follows that for x, = 5s, = D, O< y, < D 

, as eitnk(E’) 

Go(é’, Xn, Yn) = 2i (20) M Cn 7 cos(Ynk (E’)) 
K(&") 

: a eisnk(a") 

Go(0". Yns Sn) = 2i(22)"-Y¢, cos(y,K(0")) 
K(0’) 


Inserting this into (32) leads to 


D 
5@.D.o!, D)~ A | / e640) cos(ic(E) yn) Coste (Yn) £0", Yn) 
0 Rn-1 


dy'dyn, 


ei (DK(E)+DK(0")) 


A = —4c? (21)""'k?G(@) —____ 
7m)" 9) ay 
The y’ integral is a (n — 1)-dimensional Fourier transform. Hence 


D 


$(&’, D,o', D) » (20) V7 4 - cos(k(E)¥n) cos(k (0) Vn) F(E +0". Yn) dVn- 


0 
. (33) 
Here f is the (n — 1)-dimensional Fourier transform of f with respect to x’. Since 


cos(k(E’) yn) cos((k(0") yn) = xfcos((K(E') + k(0")) Yn) + 
cos((K(§") — K(0')) Yn} 


we have 
H(p', D.o!, D) 2x)" PAC FE +0, Me) + eo) 
(CIE +0’, KE) —K(0'))} (34) 


where C denotes the cosine transform: 


Fig. 10 Top: Principle of 
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(Cfy(é) = 7 cos(fx) f(a)dx. 


0 


Equation (34) is the solution in the Born approximation of the inverse scattering 
problem with a reflector. The essential difference to the solution without a reflector, 
as given in (9), is that the argument «(p’) — «(o’) shows up, indicating that small 
frequencies are involved. 

In [43] it is shown that (34) in fact permits the determination of the Fourier 
transform of f for frequencies down to 0. In [45] it is shown what can be achieved 
by a combination of reflectors with the Kaczmarz method in mammography [5, 58] 
see Fig. 10. Again we use 19 sources at the top boundary. The relaxation parameter 
a is 3 x10’. We work with a source pulse whose signature is uniform from 50 to 
500 kHz. Without the mirror the results would be completely unsatisfactory. 


8 Direct Methods 

In this section we mention some non-iterative methods which provide exact 
solutions to the nonlinear problem. 

Boundary Control 


The boundary control method for inverse problems of hyperbolic differential equa- 
tion has been developed in [1]. We follow [55] and start out from the formulation 
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(1)-(3) of the inverse problem. Let u; be the solution of (1) with du/dv = h on 
I x (0,T) and a*(t) = (a(t) + a(2T —t))/2 for any function a. Let for functions 
h,gonTI x (0,T) 


thea] = f Pda (8. Pg (x Ta 
2 
[h. g]; =f Yunls.7)-Yugle. Tas, 


The key to the boundary control method are the relations 
T 
teal= ff wzh-etuaxat, (35) 
r Jo 


T. 
oh = ff wiht etm, adr, G6) 
rJdo 


Note that the right hand sides of (35), (36) can be evaluated without knowing c. All 
we need to know is uj, ug on I” x (0,27) for the functions h, g, and this is just the 
data for the inverse problem. 

Now let g be any harmonic function in §2. We determine a function h? on I" x 
(0, 7) such that ujy = gy. Let hs(x,t) = q(t)d(x —s) for each source s on I”. We 
try the Ansatz 


h? = Yak 


with certain coefficients ds, i.e., 


g(x) = > dsuy, (x, T). 
The coefficients a, may be determined from the equations 
[Ye Yun. Todi = Dray f Yu, eT) Vu, 0. Pde 
Q 5 Q 
which, by the Gauss integral theorem can be written as 
[ Peon Prax =F aslo tl (37) 
. av hy, ’ - SL ey s Pes tT 


The left hand side and the coefficients on the right hand side are determined by the 
data. Solving (37) for as we get the function h?. We do this for many harmonic 
functions g, y. By (35) we then can compute the numbers 
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ul cpwdx = fh? ,h”). 
Q 


Since the products of harmonic functions are complete in L(£2), this determines c. 


Inverse Scattering 


The inverse problem of ultrasound tomography is a typical case of inverse scattering. 
Uniqueness questions have been dealt with in [37,52]. Since the methods in these 
papers are, in principle, constructive, numerical methods can be based on these 
methods, see [7]. 


9 Conclusion 


The mathematics of sonic imaging is well understood, at least for the linearized 
problem with constant background. Reasonably efficient iterative methods for the 
fully nonlinear problems are available for transmission imaging. Time domain as 
well as frequency domain methods are being used. In reflection imaging the main 
problem is the lack of low frequencies in the source pulse. It is not clear at present 
if this problem ever finds a satisfactory solution. 
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Abstract 

We give a self-contained presentation of coherent array imaging in random 
media, which are mathematical models of media with uncertain small-scale 
features (inhomogeneities). We describe the challenges of imaging in random 
media and discuss the coherent interferometric (CINT) imaging approach. It is 
designed to image with partially coherent waves, so it works at distances that do 
not exceed a transport mean-free path. The waves are incoherent when they travel 
longer distances, due to strong cumulative scattering by the inhomogeneities, and 
coherent imaging becomes impossible. In this article we base the presentation of 
coherent imaging on a simple geometrical optics model of wave propagation with 
randomly perturbed travel time. The model captures the canonical form of the 
second statistical moments of the wave field, which describe the loss of coherence 
and decorrelation of the waves due to scattering in random media. We use it to 
give an explicit resolution analysis of CINT which includes the assessment of 
statistical stability of the images. 


1 Introduction 


Sensor array imaging is an important technology in a variety of applications such 
as medical ultrasound, nondestructive evaluation of materials, underwater acoustics, 
geophysical prospecting, radar, and elsewhere. In general terms, it seeks to estimate 
wave sources or reflecting structures in a medium using measurements at the 
sensors, which are devices that transform one form of energy into another. In 
radar the sensors are antennas that convert electromagnetic waves to/from electrical 
signals. In underwater acoustics they are hydrophones that convert changes in water 
pressure to electrical signals, and so on. 

The array consists of many sensors which are placed close together on a 
measurement surface, so that they behave like a collective entity. It occupies a 
bounded set, the array aperture, which we denote by A. In problems like synthetic 
aperture radar [16], there isn’t an actual array. Instead, one or more sensors mounted 
on a moving platform span a long path, the synthetic aperture. To fix ideas we 
assume throughout an actual array, but most of the results presented here extend 
to problems with synthetic apertures. 

When the sensors are just receivers, we say that the array is passive. The 
sources of waves are typically far away from the passive array, and the problem 
is to determine them from the measurements at the receivers. Active arrays consist 
of sensors that are sources and receivers. They probe the medium with waves 
emitted by one or many sources and receive the echoes from the unknown reflecting 
structures. 

The recordings at the receivers are called the array data. We are concerned with 
coherent imaging systems, meaning that the data are time-resolved measurements 
of the waves. Equivalently, in the Fourier domain, the data are measurements 
of the amplitude and the phase of the Fourier components of the waves, for all 
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the frequencies in the bandwidth. Incoherent imaging systems like in infrared or 
optical tomography measure the net intensity of the waves, which are typically 
single frequency or narrow-band. They image at distances that are larger than a 
transport mean-free path, a length scale which marks the onset of diffusion in 
random media [29, 35]. They involve very different data processing and give worse 
image resolution than the coherent ones. 

Imaging is a simplified inverse problem for the wave equation. One example of 
an inverse problem is to determine the coefficients in the wave equation, like the 
wave speed, from measurements of the solution at the sensor locations. This is a 
nonlinear and ill-posed problem that is difficult to solve [30, 33, 34]. Imaging seeks 
less information about the medium, such as the loci of jumps of the wave velocity, 
which model the location of reflecting structures. It is an easier problem as long as 
we know the source of waves and how the waves propagate. 

An imaging system is useful only if it is robust to uncertainties in the problem. 
We may think of uncertainties as “noise,” but not all noise is the same. Some is easier 
to deal with, like additive measurement noise considered in most of the applied 
imaging literature. However, uncertainties in the model of wave propagation are 
much harder to mitigate. We study the effect of such uncertainties arising in imaging 
in complex media with numerous small inhomogeneities, i.e., with microstructure. 
This is unknown in practice and cannot be estimated from the necessarily band- 
limited data as part of the imaging process. The microstructure may not even be 
interesting in applications. However, it cannot be ignored because although one 
inhomogeneity by itself is negligible, there are many of them and their cumulative 
wave scattering effect builds up as the waves travel in the medium. 

We deal with the unknown microstructure by modeling it with random processes, 
and thus study imaging in random media. We describe first the forward problem, that 
is the mapping from the random microstructure to the solution of the wave equation 
at the location of the sensors. This solution is a random field. The data are one 
realization of this field, corresponding to one realization of the random medium. 
The challenge is to find imaging methods that are robust, i.e., that produce images 
which do not change significantly with the realization of the random medium. The 
robustness is called statistical stability. 

To analyze imaging in random media, we study the statistics of the wave field 
at the sensors. Specifically, we calculate the mean field, which we call the coherent 
field, and its random fluctuations, the incoherent field. When the fluctuations are 
small, the data is coherent and imaging is easier. When the fluctuations dominate 
the mean field, the data is incoherent. This is a typical regime in applications like 
diffusion tomography. We look at a regime between these two, where the data is 
partially coherent. The cumulative scattering effects are important in this regime, 
and mathematics allows us to understand them under simplifying assumptions 
like: the inhomogeneities are weak scatterers; there is separation of scales; the 
microstructure decorrelates rapidly with distance. Recent results [22,25] have 
relaxed the latter assumption to media with long range correlation, and the case 
of strong inhomogeneities can be handled in some media [21]. 
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The analysis of wave propagation in random media is quite involved, and it 
is specific to the scaling regime. There is no universal treatment of the problem, 
no one fits all regime. The detailed quantification of cumulative scattering effects 
and the calculation of important scales such as the scattering mean-free path and 
the transport mean-free path depend on the assumptions on the microstructure and 
the frequency range [35]. However, there are canonical effects such as the loss of 
coherence and decorrelation of the waves that can be given a generic description, 
via the second statistical moments of the random wave field. We describe them with 
the simplest model of wave propagation in random media, which accounts only 
for random wavefront distortions. It is a geometrical optics approximation of the 
solution of the wave equation, with randomly perturbed travel time. We use this 
model for the purpose of a self-contained analysis of coherent imaging in random 
media. The results extend to more general wave scattering regimes, as described in 
[3, 7-9, 18, 27]. 


2 Main Text 


The presentation is organized as follows: We begin in Sect.3 with the basics of 
imaging. We formulate the data model and the reverse time migration imaging 
function as an approximate least squares solution of the inverse problem. Migration 
is the basic imaging method in many applications. We describe in Sect.4 the 
challenges of imaging in complex media. The analysis of imaging in such media 
is in Sects. 7 and 8. It uses the random travel time model described in Sect. 5 and the 
setup given in Sect. 6. We show with detailed calculations in Sect. 7 how migration 
imaging fails in complex media with significant cumulative wave scattering by the 
microstructure. Then we analyze in Sect.8 the coherent interferometric (CINT) 
approach, which is designed to image with partially coherent waves. We give a 
detailed resolution analysis of this method, including the assessment of its statistical 
stability and illustrate the results with numerical simulations. 


3 Basic Imaging 


In this section we present a mathematical formulation of imaging with sensor 
arrays. We begin in section “The Forward Model” with the description of the model 
of the data recorded by passive and active arrays. We call it the forward model 
because it maps the source distribution or the reflectivity, the unknowns in the 
imaging problem, to the wave field measured by the sensors. In general there is 
no explicit inverse mapping from the array data to the unknown source density or 
reflectivity. We show in section “Least Squares Inversion” how we can state the 
imaging problem in variational, least squares data fit form. The mathematical model 
of the time reversal process can be viewed as an approximation of the solution of 
the least squares problem, as described in section “The Normal Operator and the 
Time Reversal Process”. It can be computed and used for imaging only when we 
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know the medium through which the waves propagate. We discuss its refocusing in 
smooth media and robustness to additive noise in section “Imaging in Smooth and 
Known Media.” 


The Forward Model 


We base the forward model of the array data on the scalar wave equation for the 
acoustic pressure p(t, X) 


1 0 p(t,x) 


2a a AP) = F(t,x), xeR, t>0, (1) 


where c(x) is the wave speed and F(t, x) the source density. We call p(t,x) the 
wave field and relate it to the source using Duhamel’s principle 


is 
i= i as [ Pa gCR CC / dj F(t,3)1GU,3,%). (2) 
0) R” R” 
Here G(t, x, y) is the causal Green’s function, the solution in the sense of distribu- 


tions of 


1 #&G(t,%,y) 
C(x) OL? 


— A;G(t,x,y) = (k—y)d(t), X,yeR", t>0, (3) 
G(t,x,y)=0, ¢ <0, (4) 


where Ax is the Laplace operator in the x variable, and 6 is the Dirac distribution. 
The second equality in (2) is because both the source and Green’s function are causal 
(supported at time ¢ > 0), and we can write 


t foe) 
/ ds F(s,y)G(t —s,%, 9) = ds F(s,y)G(t — 8, ¥,X) = F(t, y) *: G(t, XJ). 
0 —oo 


Convolutions are not convenient for the analysis, so we often work in the Fourier 
(frequency) domain. The Fourier transform of the wave field is defined as 


lo, ) 
plw.%) =f dt plt.ie'™, (5) 
—oo 
and the inverse transform is 


. ak.) eee meee 
pit.) = | 3, Plo we S (6) 
—oo 2 


1284 L. Borcea 


The dual variable to time ¢ in these transformations, denoted by w, is the angular 
frequency (measured in radians per second). It is related to the frequency (measured 
in Hz) by a factor of 277. We also relate w to the wavenumber k and the wavelength 
A using a reference wave speed scale cy, 


@ 20 


(7) 


The advantage of working in the Fourier domain is that the convolution in (2) 
becomes a product 


po,%) = : a} Fo, C,3,9, (8) 
R” 


with Glo, y, x) the Fourier transform of the causal Green’s function. This is the 
same as the outgoing Green’s function of the Helmholtz equation. 


Passive Array Data Model 
When imaging with passive arrays, F (t,x) is unknown. We assume for simplicity 
that it has the separable form 


F(t,x) = fp), (9) 


where the same signal f(t) is emitted from all the points in the support of the source 
density p(x). In some applications like synthetic aperture radar, f(t) is a complex 
valued and long (chirped) signal. But in other applications f(t) is a function of 
small temporal support, a pulse. We assume henceforth such a pulse and model 
F(t) as a base-band pulse f(t) modulated by an harmonic signal at carrier (central) 
frequency w,/(27), 


f(t) =e fp(t). (10) 


The terminology becomes clear in the frequency domain, where 
A oo * A 
f(@)= / da f(tye'?’ = fz(w— ap). (11) 
—0o 


The signal fg(t) is called base-band pulse because its Fourier transform fe (@) is 
supported at w € [—7B, 2B]. Then, the frequencies w/2z in the support of f (w) 
lie in the interval centered at w, /(27) of bandwidth B. 

The WN, receivers located at X, in the aperture A of the array measure the data 


D(t) = {d,%)}, t€ OT], %€A, r=1,...,N,, (12) 


over the time duration T. Equivalently, in the Fourier domain we have 
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The goal of imaging is to estimate from these data the support of p, which is 
contained in a compact set J, C R” called the imaging window. 

We use a terminology borrowed from the seismic imaging literature, and refer 
to d(t,X,) as time traces, to emphasize that they are time recordings. Their model 
comes from (2), 


S © dw BAe, Seon. Ss ee Dp 
40.8) = bant0| f Sf i Fw.D6W%.He aco]. 
—oo Ty 


(14) 
where we added the noise \V,(t) indexed by the receiver. We assume as is typical 
that NV, (t) is white noise, i.e., a real-valued stationary Gaussian generalized process 
with zero expectation, covariance equal to the delta distribution, and independent 
over the receivers 


DIN (HN) = 0, ELINA (ONY ()] = 0 6(t = 1) 07. (15) 


Here E denotes statistical expectation; 6,,/ is the Kronecker delta symbol; the 
parameter o,, scales the noise level and the time window 1(0,7;(t) is equal to one 
when t € (0,7] and zero otherwise. The waves are transient and thus can be 
captured for large enough 7. We assume such a T so we can ignore the window 
1(o,7](¢) in the first term of (14), and obtain the following relation in the Fourier 
domain, 


T 
diw.%) ~ | dy F(w, ¥)G(w,%;.¥) + NZ (o), Apo) = [ dt N, (tei. 


(16) 
The forward map M takes the unknown source density p to the data space D. We 
show in the next section how to formulate the inverse problem as an optimization 
that computes an “approximate inverse” of M in some sense. This is not an actual 
inverse which may not even exist. For optimization, it is convenient to work in 
function spaces with inner products. Thus we assume that p(y) lies in L*(/,,), the 
Hilbert space of real valued and square integrable functions supported in /,,,, with 
inner product 


in = | dyp(y)n(y). VpneLUy), (17) 


and norm ||p||2 = ./(p, p). The data lies in the vector space of functions of finite 
energy 
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N, 
0.8 6.65: de |d(w.%,) 


lo—@o|<1B 


= 
<0o;7 , (18) 


r=1 


endowed with the complex inner product 


N, 
(i.8)= f | do Lido.) R0.%), Vd. FED, (19) 


r=1 


and the norm \|d lp = (4, d ) 


The forward map M : L?(J,,) > D is the linear operator defined by 


MAC2= 76) if d¥ pG)C.%,.3). (20) 


It assumes that we know the pulse and the wave speed c(x), i.e., the Green’s 
function. The latter assumption turns out to be critical, and it is not satisfied in 
complex media with microstructure. 


Active Array Data Model 
To state the model for active arrays, we redefine the coefficient in the wave equation 
as 


1 1 p(x) 
c2(x) —c?(x) C2? 


(21) 


where c(X) is the assumed known wave speed in the medium which hosts the 
perturbation (reflectivity) o(x) that we wish to estimate. It is supported in a compact 
set J, C R” that defines the imaging region. The name reflectivity suggests that we 
expect it to cause reflected waves (echoes) that can be measured at the array. This 
is relevant for most imaging setups where the array lies on one side of the medium. 
For example, in exploration geophysics, the receivers lie on the surface of the earth 
and do not see waves that interact with the subsurface unless p reflects them. 

Suppose that the excitation comes from a source in the array, idealized as a point 
at x, € A, 


F(t,x) = f(0)5(% — Xs). (22) 


The array has N, => 1 sources which may emit simultaneously, but we assume here 
that the excitation is with one source at a time, and we emphasize with the notation 
P(t, X, Xs) the dependence of the wave field on the source location. It satisfies 
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1 0 p(t, X, Xs) 
ag op 


_ px) 0? p(t. X, 5) 
2 at? 


Ax P(t, X,%s) = (23) 


S(O) — Xs), t > 0, 


with initial condition 
ron ) i 
p(0,x, Xs) = a, PO. x, x;) = 0, (24) 


and we can write it in terms of the causal Green’s function using Duhamel’s 
principle 


PUK. %) = flO) #1 GR a) ff dj eee + G(t%.9). (25) 


In the forward model we evaluate (25) at the receiver locations xX = x, € A, 
forr = 1,...,N,. (The sources and receivers may be collocated). The first term 
S(t) *; G(t, X;, Xs) is the direct wave that does not interact with the reflectivity. We 
can remove it by time windowing, assuming that it arrives much earlier than the 
echoes from the medium. Thus, we redefine the origin of time of the measurements 
after the direct arrival of the wave from X,, and let henceforth p(t, xX,,X;) model 
the echoes from the medium for t € (0,7]. The second term in (25) depends 
nonlinearly on p. We use its linearization, known as the Born approximation, as 
is typical in imaging. The approximation holds when p has small amplitude and/or 
small support, although numerical experiments suggest that it may have a wider 
range of validity. We refer to [32] for its analysis in media with general bounded 
background wave speed c(x). 

The Born data model is 


— ~- dw A edt a een ce Seas ae 
Hee Bik / See) / dFpHE(w,%,, PE,F, Fett + 
be ; 


lor ON;s(t), (26) 
where NV;;(¢) is the additive noise indexed by the receivers and the sources. The 


latter index reminds us that the noise varies from one illumination to another. We 
suppose as before that \/,,(t) is white noise, independent over the array sensors 


{ [Nis (t)] = 0, D [Mis (1)Ny/s" (t’)| = oa. 8(t _ t')8y7/85,5! : (27) 


We also choose a large enough 7, so that we can neglect the effect of the window 
in the first term of the right-hand side of (26). 

The forward map M : L?(,,) — D takes square integrable reflectivity functions 
p to functions in the data space 
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Ny Ns 
y= }dlo.se et, > dw \d (@,%.%) e of , (28) 


|o—wo|<2B 


r=1 s=1 
with complex inner product 


Nr Ns 


(Z.a)= [Ao Vd@.% H)O.%,%) Vd.geD, (29) 


r=1 s=1 


and induced norm \|d lp = (4. d \ It is given by 


Mie t2=2 eo) / dF0DEW,%, NEO, 9, %), (30) 


and it assumes that we know both the pulse a (@) and the Green’s function, 1.e., 


(x). 


Least Squares Inversion 


The imaging problem is to estimate from the array measurements the location of 
the unknown remote sources and/or reflectors in the medium. We show here how 
it relates to the problem of least squares minimization of the misfit between the 
measured data and the model prediction. Because the data space depends on the 
type of the array, we treat separately passive and active arrays. In both cases the 
unknown lies in the Hilbert space L7(/,,) with inner product (17). 


Connection to Bayesian Inversion 

The least squares minimization of the misfit between the measured data and the 

model prediction is widely used in the literature. It arises in Bayesian inversion, 

where the least squares minimizer is known as the maximum likelihood estimator. 
Denote by z(d |p) the density of the likelihood function, which is the probability 

that we observe the data d given p. Bayes’ rule says that the posterior probability 

density 2(p|d) of p, given the data, is 


m(d|p)x(p) 
m(p|d) = ad)” 
where zr(p) is the prior probability density of p and z(d) is the marginal likelihood 
density of the data. Assuming no information about p prior to the measurements, 
we take a uniform density z(p) over the imaging region. The marginal likelihood 
z(d) tells us if the data can be achieved by measurements. We suppose that this is 
so, which means that z(d) is just a constant which plays no role in the estimation. 
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Under all these assumptions, we obtain that maximizing the posterior 2(p|d) is the 
same as maximizing the likelihood function 2(d|p). This is given by 


(31) 


ld — Mell; 
m(d|p) ~ exp at ’ 
Or 


and its maximizer, the maximum likelihood estimate, is the same as the least squares 
solution described in the next sections. The symbol ~ in Eq. (31) means equal up to 
a normalization constant. We have a normal probability density because we assume 
white noise which is Gaussian and independent over the receivers. The variance 
oe = To, is the noise power at a sensor, calculated over the duration T of the 
measurements. The noise is essentially uncorrelated over the frequencies when the 
bandwidth is sampled at intervals Aw >> 27/T, as follows by a straightforward 
calculation. 


Imaging with Passive Arrays 
The variational formulation of the inverse problem is: Find p € L?(J,,), a minimizer 
of the data misfit 


A Nr im 2 
O0)=1d-Molh= [do Y|A@.%)—Mal@.%)[ - 62) 


r=1 


We may add a penalty term in (32), such as a multiple of ||p||5 to regularize the 
problem. We do not do so because it does not play a role in the basic imaging 
methods derived below. However, regularization is important and one can use other 
penalties, such as total variation and sparsity promoting norms, to improve the 
quality of images [15, 26]. 

The first-order optimality condition satisfied by the minimizer of (32) is 


IM*Mp] (§) = [Mid |), V¥" Ee Iy, (33) 


where the operator M* : D + L?(J,,) is the adjoint of M, defined formally by the 
relation 


(@.,.Mp) = (M'd.p), VpeL,), deD. (34) 


It is given explicitly by 


N, 
[Md]@ =f dw fo LA.%)GW.H.%). 5) 


|o—ao|<7B pel 
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where y* are the search points that sweep the imaging region /,,, and the bar denotes 
the complex conjugate. 

Equation (33) are the normal equations and M*M : L?(J,,) > L°(/,,) is the 
normal operator 


IM* Mp] (§°) = ; dF PG) Kas 9.F"), (36) 


with kernel 


A 2 yr a 
KG) =f | do [Fo Y*G(o.9.%)G(0,¥.%). 37) 


r=1 


It is usually not invertible, so one way of solving (33) is to compute the pseudo- 
inverse of M* M, which is typically very expensive. However, the kernel Kpas(y, y°) 
has the important property that is large for points y in the vicinity of y° and small 
otherwise, as we explain in more detail in section “The Normal Operator and the 
Time Reversal Process.” Thus, from the point of view of approximating the support 
of p, we may replace the normal operator in (33) by the identity, and obtain that 


N, 
pg) ~ [Md |@) = / ep OPO) VL AO.K)GW.F.%)- G8) 
O-Wo |< 7=1 


The factor 7 (@) is not essential in imaging with pulses, although it is important 
when the source emits a long chirped signal or a stationary noise signal. In that case 
the factor ic (@) arises in a data pre-processing step known as pulse compression. 
Since we assume that the source emits a pulse, we may neglect f (@) in (38) to 
obtain 


dw 


N, 
ae) ~ TF) = / 2 de.%) (0, %)G (0.9%). 9) 


lo-@,|<7B 20 


This is useful for applications where f(t) may not be known. The normalization 
constant 1 /(27r) is convenient for inverting the Fourier transform. 

The meaning of the symbol ~ in (38) and (39) is that large values of the 
right-hand sides correspond to points in the vicinity of the support of p. Thus, 


[med | (y’) or its simplification 7 (y*) are imaging functions. 


Imaging with Active Arrays 

The case of active arrays is similar to the above, except that the data space is given by 
(28), and the complex inner product is (29). The unknown reflectivity p is estimated 
by a minimizer of 
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d(a, X;, Xs) — [Mop] (a, 225) 


O16) = 1d - Mol = | wry 


O—Wo|S2B r=1s=1 
(40) 
and satisfies the normal equations 
[M*Mpl (9) = [Md] @), V¥* € Ty. (41) 
The adjoint operator M* : D — L*(J,,) defined formally by 
(a . Mp) = (rd .p), VpeL*(Iy), d€D, (42) 


has the explicit expression 


N, Ng 
[mid] @) = “ean OPPO LY M0 HOOF %) 48) 


r=1 s=1 


and the normal operator M*M : L?(1,,) > L?(1,), 
MMA] ®)= [a9 G)KuG.F. (44) 
Ww 


has the kernel 


x 2 7 a ee a 
Kac(¥, 9°) = | dw k* MO) YS > G.Y,%,)G(o, ¥', %) 


O-—Wo|<7B 7=1 


= 


G(0,¥,%s)G(W, 9°, %). (45) 
1 


ll 


We discuss this kernel in some detail in section “The Normal Operator and the 
Time Reversal Process,” and we relate it to the kernel (37) arising in imaging with 
passive arrays. Again, we expect that K’,.¢ peaks near its diagonal, so we can estimate 
the support of the reflectivity by replacing M*M with the identity in the normal 
equations. We obtain 


N, Ns 
e@)~[mrd]@ =f dor? FOL LAO%.%) oe acws Ss) 


|o—ao|<xB 


G(w, y", Xs), (46) 
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with symbol ~ having the same meaning as in (39). 

The factor w? f (w) in (46) amounts to convolving the second derivative f” (t) 
of the emitted signal with the time reversed data. But in the Born approximation 
(26) the time reversed data is determined by f”(—t) convolved with the Green’s 
functions. Thus, we have f” (t) x, f’”(—t) in (46). Such convolutions are important 
only when the emitted signals are long, like chirps, or noise signals. Because our 
f(t) is a pulse, we can estimate the support of the reflectivity using the imaging 
function 


Nr Ns a 
S> Yo d@,%.%s)G(0.¥'.%) (47) 


r=1 s=1 


a 7 d 
0G) ~ JG) = i = 


lo—o|<xB 21 


The Normal Operator and the Time Reversal Process 


We draw here an analogy between the normal operator and the time reversal process, 
to give a physical meaning of its replacement by the identity in Eqs. (38) and (46). 
The time reversal process is described in section “The Time Reversal Process.” The 
connection to the normal operator is in section “The Normal Operator.” 


The Time Reversal Process 

The time reversal process is a physical experiment that uses an array of N sensors 
which act as both receivers and sources. It is a two-step process. In the first step 
the sensors are receivers which record over a time window y7(t) of duration T, 
the waves generated by a remote point-like source at y that emits a pulse g(t). This 
may be the same as f(t), but it is convenient for the discussion to take an arbitrary 
function g(t), with the same bandwidth as f(t). We can model the density of the 
point-like source by letting p be supported in a ball B, (¥) centered at y of radius 
R < Aq. This allows us to approximate the Green’s function in the model of the 
data by its value at y, and obtain a net source amplitude equal to the integral of 
p over B,(¥). The wave equation and the imaging function are linear in p, so we 
normalize the net source amplitude to one. 

In the second step of the experiment, the array time reverses the recordings 
and re-emits them simultaneously from all the sensors, which act as sources. The 
resulting wave propagates in the medium, and it refocuses near the original source, 
due to the time reversibility of the wave equation. The quality of the refocusing 
depends on how large the array is, the length of time of the recordings, and the 
medium. 

The recordings at the receivers are denoted by 


at;%,) = Xr (t)e(t) ay G(t,X,,Y), (48) 
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and their time reversed version d,,, (t,X,) = d y(T -t, X,) has the Fourier transform 


ding (@,%,) = e!°T dy(w,%,), (49) 


forr = 1,...,N. This is the ideal model, without additive noise. We obtain from 
(48) that 


A . dw’. . A ar : A Ss 
b(G,.3)) = / 5, Aw'T)G(@ — w')G(w — w',X,, ¥) © G(@)G(w, X,. ¥), 
(50) 


where we let 


1 
Xr) = aes (+) : 


for a function x of dimensionless argument and support of order one. The 
approximation in (50) is for large recording times T satisfying w,T >> 1, and for 
the window normalization 7, (0) = 1. 

The array emits the time reversed field d,, from all the sensors at once, and the 
wave observed at the search points y* is modeled by 


N 
25 2 dw “ > A > 3s -—iw 
Pp +t.y.y) = y [| Pan 560.5. Fe (Tt) 
r=1 


ll daz. 
= yy) —d,(o,X,)G(o, x,y Je 
= 20 


N 
d Ae Sa SS ee = _ 
~ a 5 PONG (0, %,, E(w, %,.F)e ae (51) 


r=1 


where we used the approximation (50). Because of the time reversibility of the wave 
equation, it refocuses at the original source, at offset time t = 0. The time reversal 
point spread function 


N 
ss 2 m5 2 dw—~> s a A > Bs 
Kir) = Pre(T. yy) © ) [ PFO E0560 5.9 (52) 


r=1 


models the refocused wave field. We describe it briefly in section “Imaging in 
Smooth and Known Media” in the case of smooth media. The cross-range resolution 
of the refocusing depends on the wavelength, the distance from the source to the 
array and the array aperture. The range resolution is determined by the bandwidth. 
In complex media the focusing may be improved. This is known as super-resolution. 
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By improved we mean that with a limited aperture we may get better cross-range 
resolution than we would if the medium were smooth. We refer to [6, 19,27] for the 
analysis and illustration of super-resolution. 


The Normal Operator 
In the case of passive arrays, the normal operator is given by (36). It is a linear 
integral operator with kernel Kpass(Y, y°) written explicitly in (37). We see that it is 
the same as the time reversal point spread function (52) for g(w) = | 7 (w)|*. That 
is to say, the kernel of the normal operator is mathematically equivalent to the time 
reversal function for a source at y, emitting the pulse with Fourier transform equal 
to | f (w)|*. The tighter the refocusing of the wave at y in the time reversal process, 
the more the kernel is peaked at y = y’, and the closer the behavior of the normal 
operator to an approximate identity in Eq. (38). 

The normal operator for the case of active arrays is given by (44), and its kernel 
is (45). Let us write it assuming that each sensor in the array acts as a source and 
receiver, so that VN, = N, = N. We have 


N 


2 
Kael(¥. 9°) = i | dori] Ga (0%. NG(W.F' 5 co (53) 


which can be related to the time reversal process using the time-dependent field 


N ——————— 
PMT HEP =f dof) EO.%.DCOF. RI, 


lo—@9|<7B 
(54) 
This is the wave observed at location y* and time T + f in the time reversal process, 
in the case of a source at y, emitting a pulse with Fourier coefficient k7| 7 (w)|. The 
focusing in (54) is expected at tf = 0, and the time reversal point spread function is 
equal to pw )(T, y, y°). The kernel (53) is, up to a constant, the time convolution of 
ps ) with itself, evaluated at t = 0, 


r=1 


Kaa. ¥°) = 2p (T + 1,9, 9°) «1 POT + LY.) |, 20: (55) 


If the wave field p\")(T + t,¥,¥*) focuses at ¥¥ = y around +t = 0, so will the 
kernel. Thus, the normal operator behaves as an approximate identity if the time 
reversal process focuses with sharp resolution. 

The generalization of (55) to arrays with N, receivers and N, sources is 


Kact(¥. 9°) = 2ap™(T +, 9. 9°) # p(T +. 9.9°)|_9- (56) 
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In particular, for a single source N, = 1, and there is no refocusing of p&o(T + 
t,y, y°) at y’ = y. The behavior of the kernel is determined by the refocusing of the 
wave p&o(r +t,y.y’). 


Imaging in Smooth and Known Media 


In known media the imaging function (39) is the mathematical analogue of the 
time reversal point spread function. We describe it in this section for the case of 
smooth media. In unknown media we cannot compute (39). We use instead the 
reverse time migration imaging function that propagates the waves from the array 
to the imaging region in a surrogate medium. It is the method of choice in many 
imaging applications. The Kirchhoff migration imaging function is a high frequency 
approximation of the reverse time migration function, where the propagation is 
replaced by a synchronization of the waves using travel times. 

Once we have defined the imaging functions, the question is how well do they 
work? To answer it, we identify two properties that make an imaging method useful: 


1. The image should focus at p, i.e., |.7(y*)| should be large near the support of p 
and small elsewhere. 
2. The focusing should be robust. 


We study in section “Robustness to Additive Noise.” We postpone the more involved 
discussion of robustness to uncertainties in complex media until Sects. 7 and 8. The 
characterization of the focusing of the image is known as resolution analysis. We 
state the resolution limits for a setup with small arrays. 


Passive Arrays 

When we know the medium, and therefore the Green’s function G, we can process 
the data as in (39) to form an image. The data processing has the physical 
interpretation of taking the traces received at the array, time reversing them, and 
then back-propagating them to the imaging point via the Green’s function. Because 
the Green’s function models the actual wave propagation in the known medium, the 
back-propagation in 7(y*) is equivalent to solving the wave equation with source 
term 


Ny 
F(t,%) = ) | d(-t,%,)5(% —%), 


r=1 


and evaluating the pressure field at y’ € J,,, at the expected time of refocus. The 
imaging process is mathematically equivalent to the time reversal process described 
in section “The Normal Operator and the Time Reversal Process” because we know 
precisely the medium! 
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To study the imaging function (39), let us recall the model (16) of the data and 
write 


N, 
76) = | | a0? S > do. %)G(o, ¥'.%,) =(TH)) + 87H). (57) 


r=1 


The first term is the model of the image in the absence of noise 


N, 
(1G) = | a9) / _ FOLGER DGOF%), 68) 


r=1 
and the second term models the effect of the noise 


Ny | 
sr@) = | do Y NF (o)G(,$",%). (59) 
|o—@o|<7B r=1 
We study 67 (y°) in section “Robustness to Additive Noise,” so we concentrate here 
on (J (y*)), which we rewrite as 


fa ae ga ie 2 sy iw[t(X;.y°)—t (X.Y 
(7(9°)) ~ | aso@) | agi De Fo) Ya, Hak, Fei H HI 
Ww @O—-Wo|< r=1 
(60) 
Ny 
= / d¥p3) Ya Hak. ¥°) f (eG. H) — 1.9). (61) 
w r=1 


Here we used the geometrical optics approximation of the Green’s function in the 
assumed smooth medium 


G(w,%,9) © a(%, Pel" | (62) 


with amplitude a and travel time t. They are calculated along rays (geodesics) 
connecting xX to y by solving with the method of characteristics a transport equation 
and an eikonal equation [5,24]. In the case of homogeneous media with wave speed 
Co, they have the explicit expression 


a 3 as 1 3 as = 
a(X, Y) = W(X, y) = 4n|k—¥| and 1(X,y) = 1)(X,y) ‘= —-* (63) 


We can understand what to expect from (61) by referring to the example in Fig. 1, 
with a source of small support around point y. Because data d(t, X,) are essentially 
the emitted pulse delayed by the travel time 1(X,, y), we get a contribution to the 
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100A, 


100A, 


Fig. 1 Plot of the passive array data traces for a point source in a homogeneous medium. They 
solve the wave equation in a homogeneous medium with wave speed cy = 1.5 km/s, for a source 
emitting a pulse with wide frequency band [1.5-4.5] MHz. We plot d(t,x,) in grey scale, as a 
function of time in the abscissa and receiver location in the ordinate. All the lengths are scaled by 
the central wavelength A, = 0.5 mm. The simulation setup is shown on the left 


Fig. 2. Kirchhoff migration 38 
image for one point source in 

a homogeneous medium. This 40 
is the image obtained using 

(64) for the passive array data 49 
traces shown in Fig. 1. We 

plot the absolute value of the 44 
imaging function. The 

abscissa is range in, and the 4¢ 
ordinate is cross-range in A, 


48 
50 
52 
54 
56 


58 


sum in (61) if y° lies near the isochrone with travel time equal to t(X,, y), at least 
for one receiver. For example, in homogeneous media this means that y* is near the 
sphere of center x, and radius |X, — y|, at least for one receiver. With this reasoning, 
we expect that (a: (y°)) is large near the intersection of all these isochrones, for all 
receivers in the array. Such points are close to y, as we illustrate in Fig. 2. The 
imaging function for point-like sources is called the point spread function and its 
essential support around y defines the resolution limits. 

We assume in this article a typical imaging regime with an array of linear size a 
that is much smaller than the distance to the source, and with bandwidth B < a,j. 
The direction from the source to the array is called range, and the distance along it 
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is of the order of the range scale L. The directions orthogonal to the range are called 
cross-range. The support of the point spread function in cross-range, the cross-range 
resolution, is of the order A, L/a. It improves as we increase the frequency and the 
array aperture. The range resolution depends on the temporal support of the pulse, 
which defines the precision of the arrival time estimation of the waves, and thus of 
the distance to the source. Because the pulse f(t) has the temporal support O(1/B), 
it is of the order c,/B. These resolution limits are widely known and can be deduced 
from the calculations in Sect. 7. 

A common imaging approach in the applied literature is a further simplification 
of 7” (¥°), known as Kirchhoff migration [4], 


N, 
I" O)=> 2G) =): (64) 


r=1 


It is similar to reverse time migration except that it neglects the geometrical 
spreading factor a(y*,x,) in the Green’s function, as if it were a constant across 
the array. This approximation is justified in our setup because the array aperture is 
much smaller than the distance between the array and the imaging region. 


Active Arrays 
The reverse time migration function for active arrays is 


alc / YY d(0,%,,%,)G(0,%,, F)G(,%F) 


lo—o,|<1B uv 


_— 21 


N, Ns 


= S55 dG, ¥°) + 1%.) Kr, Ks JAK, VaR, ¥’). (65) 


r=1 s=1 


iS 


d (W,%;, JOR, P)aK,, Pel H+ GH] 


r=1 s=1 


The Kirchhoff migration function neglects the geometrical spreading factors 
a(x;,y°) of the Green’s function, and forms an image by simply summing the 
traces “synchronized” by the expected round trip travel time, 


N, Ns 
TH) = LY de. F) + 1K. HF"), KH). (66) 


r=1 s=1 


We show in Fig.5 the Kirchhoff migration images obtained with the data traces in 
Figs. 3 and 4. In each case we indicate the location of the scatterers with a black 
circle (Fig. 5). 
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Fig. 3 Plot of the active array data traces for a point reflector in a homogeneous medium, and 
an illumination from the center element in A. The abscissa is time in «vs and the ordinate is the 
receiver location in A,. The simulation setup is shown on the /eft. It is the same as in Fig. 1 with 
the source being replaced by a soft acoustic reflector, i.e., a disk of diameter A, with homogeneous 
Dirichlet conditions on its boundary 


100A, 


8 86 8 8 


100A, 


Fig. 4 Plot of the active array data traces for three reflectors in a homogeneous medium, and an 
illumination from the center element in the array. The abscissa is time in «ts and the ordinate is the 
receiver location in A,. The simulation setup is shown on the /eft. The Born approximation used 
in the derivation of the imaging function captures only the primary echoes shown by the strong 
hyperbolae in the figure. The later arrivals are multiply scattered waves between the reflectors 


We leave the interpretation of the imaging functions to the reader. They are 
straightforward extensions of the results for passive arrays. 


Robustness to Additive Noise 


To quantify the effect of the noise, we view the imaging problem in a stochastic 
framework, where .7(y*) is a random process due to the noise. The expectation of 
J(¥*) is its coherent part Ae (y*)] = (7(y*)). and the random fluctuations are 
5T(¥') = Jy’) -—E[7(¥*)] . Intuitively, robustness of the imaging method means 
that the coherent part (.7) dominates the fluctuations 6,7 in the vicinity of its peaks, 
so we can determine with high fidelity the support of the unknown p. That is to 
say, the images do not change in essential ways with the realization of noise. In an 
imaging experiment we can compute only one realization of the random process 
J(y*), which is what we call an image. It corresponds to the one realization of the 
noise in the data gathered by the receivers in the array. If we could compare such 
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Fig. 5 Imaging results in homogeneous media using the traces plotted in Figs. 3 and 4. The images 
are computed with Eq. (66). The absolute value of the image for one reflector is shown on the left 
and for three reflectors on the right. The abscissa is range in A, and the ordinate is cross-range 
in A,. The imaging function is based on the Born approximation, and the image has some faint 
artifacts due to the multiply reflected waves seen in the right plot of Fig. 4 


an image with (J (y*)), the ideal image with noiseless data, we would see many 
spurious peaks distributed throughout the search domain J,,, i.e., one realization 
of 57 (y*). If |8.7| is smaller than | (7) | for points in the support of p, with high 
probability, then the estimates of the support of p are essentially independent of the 
realization of the noise and the method is robust. 

We assess below the robustness of images with passive arrays. The case of active 
arrays is similar. Recall Eqs. (57)—(59) and the assumptions (15) on N,.(t). Assume 
also a point-like source at y in order to simplify the calculations. The coherent part 
of the image evaluated at y is 


Nr Nr 
dw FO) D|6@.%.9) ~ [OD la&.H?, 67 


r=1 r=1 


(J(¥)) © / 


|o—@|<7B 


and we wish that its magnitude be larger than |5.7(¥)| with high probability. We can 
estimate this probability using Chebyshev’s inequality 


_ ELLIO] 


PLIT@ > \(87@H)|] =1- P[IH| = (7) ] = 1 
[7M > (57M) [BIH = (7M) (7@)F 


1 2 
='-(aaay) 


(68) 
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where SNR stands for the signal to noise ratio 


snr@ = ZO Var [ 7(y)] = VE[|67()/7]. (69) 


Var [.7(¥)| 


The “signal” is the coherent part (67) of the random process 7(y) and the “noise” 
is 5.7(y). Its model follows from (59) and the geometrical optics approximation of 
the Green’s function 

Ny 

579) ~ DY al&,.¥°) da NT (weit), (70) 

vet |o—ao|<7B 
It is a linear superposition of independent Gaussian noise terms, so it is Gaussian, 
with mean E [87 (y*)| = 0 and variance 


var [I (§')] © 03, Slog eff den dey! el0-0")"B¥) 
lo- o|<1B 
i dt e ie wo’ )t 
~ 41° Box > la, ¥°)P. (71) 


r=1 


The last approximation is for very large T and for Bt(X,,y°) >> 1, as is the case 
under our assumption that the temporal width O(1/B) of the pulse is much smaller 
than the travel times from the source to the array. 

The SNR follows from (67) and (71) 


| £(0)| AFL |e ep 
———— ) Ss ; 72 
210,/VB fa ( ) 


= 2n0,, 
r=1 


SNR [ 7(¥)] ~ 


Se y| < 


r=1 


where 


IP = f atlf@P = do| flw)P. 


|o—@o|<1B 


It increases linearly with the norm of the emitted signal and decreases with the 
“noise level” o,;. Having more sensors is beneficial because the sum over r 
increases. The amplitudes a decrease with the distance from the source to the array, 
so the noise level should be much lower than the norm of the emitted pulse in order 
to have high fidelity images of remote sources. 

We neglect henceforth additive noise, for simplicity. Noise is of course part of 
any imaging experiment. It can be added easily in the analysis that follows and its 
effects can be studied as described above. 
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4 Challenges of Imaging in Complex Media 


In the previous section we considered media that were largely known and smooth, 
with the exception of a compactly supported embedded reflector. In many imaging 
applications with active arrays the medium is complex, with wave speed c(x) that 
is not smooth and unknown. A natural idea is to estimate the function c(xX) by the 


a 2 
minimizer of the least squares functional |z (c)-—d |. , where F is the nonlinear 


forward mapping that takes the wave speed c to the data space D. Given the 
large number of unknowns, corresponding to a discretization of c(x) on a grid, the 
optimization would have to use a gradient-based optimization method like Gauss— 
Newton [23], which linearizes locally F(c). It turns out that it is easier to linearize 
F (c) with respect to localized perturbations of c(X) than with respect to distributed 
(smooth) ones. For example, the smooth part of c(x) determines the travel times 
t of the waves, and in high frequency regimes small perturbations of t give large 
changes of the wave field that oscillates like e’®*. This indicates that it should be 
easier to estimate a sparse set of large discontinuities of c(x), which is the goal in 
imaging, than the rest of the function. 

There is computational experience on the least squares minimization over c(X) 
in piecewise smooth media, and some convergence studies exist [30, 31, 34, 37]. 
The problem is difficult because the least squares functional is not convex, and the 
optimization gets stuck easily in local minima that have nothing to do with the true 
c(X). Thus, success can be expected only when the initial guess of c(X) is very good. 
What this means exactly is complicated and poorly understood at the moment. 

The most powerful idea that has emerged in inversion in complex media is that of 
separation of scales. We already introduced it in Eq. (22), where we distinguished 
between the background speed c(x) (assumed smooth there) and the reflectivity 
p. We extend it in the next section to media with microscale. Reverse time 
migration is designed to estimate p, and as we explained in the previous section, 
it is based on the linearization M of the forward mapping ¥. This linearization 
works well in many cases, but when there are strong reflecting interfaces like air- 
solid boundaries, the data contain strong multiply scattered echoes which cause 
image artifacts. See for example [36] for data processing algorithms intended 
to suppress such multiple reflections. In any case, migration imaging can work 
well only when we know the background velocity, which determines the wave 
propagation between the reflection events. When we have it wrong, the migration 
images are not well focused and the reflectivity is mapped to wrong places. The 
estimation of the background speed is called velocity analysis or inversion. We do 
not discuss it here, but we note that it is strongly connected to migration imaging 
[4]. On one hand, the quality of the migration images depends on knowing the 
background speed, and on the other hand this speed can be estimated by comparing 
migration images formed with various data subsets [14]. This idea is behind the 
iterative migration-velocity analysis inversion methods that are popular in fields like 
reflection seismology [4]. 
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Cluttered Media 


Imaging becomes complicated in media with rapidly fluctuating wave speed c(x) 
due to numerous small inhomogeneities. Because the array data is necessarily 
band limited, it is not possible to recover all these fluctuations in detail as part 
of the inversion. This motivates us to separate the wave speed in two parts: 
the “macrostructure” which can in principle be recovered in inversion, and the 
“microstructure” or “clutter” which is the uncertain part of c(x). What we call 
macrostructure is the smooth part G(x) of the wave speed assumed known, and 
a sparse set of its large discontinuities modeled by the reflectivity p(x). Strictly 
speaking the reflectivity is a fine scale feature of c(x), but it can be estimated from 
the data so we think of it as part of the macrostructure. The assumption that Cc is 
known is because we do not consider velocity estimation. We refer to [17] for an 
example of robust velocity estimation in complex media with microstructure. 
The separation of scales is now 


paul te()® os 
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with j(X) representing the uncertain microstructure or clutter, the rapid fluctuations 
of c(X) on scales £ comparable to the wavelength or less than that. The amplitude 
o of ,2(X) is typically small, meaning that an inhomogeneity by itself is a weak 
scatterer with respect to p. It is the cumulative effect of the inhomogeneities 
that becomes significant as the waves travel deeper in cluttered media. See for 
example the illustration in Fig.6, where we have the same setup as in Fig. 4, 
but the wave speed has rapid fluctuations, as shown on the left. The amplitude 
of these fluctuations is small in comparison with the reflectivity of the three 
point-like scatterers modeled in the simulations as “sound soft,’ with the acoustic 
pressure vanishing at their boundary. These three scatterers cause strong echoes 
that are somewhat visible in the noisy traces, with arrivals lying on curves that are 
perturbations of the hyperbole in Fig.4. But there are many other echoes recorded 
before and after these arrivals, due to multiple scattering by the inhomogeneities. 
They are more significant than the waves scattered between the reflectors in Fig. 4, 
so we will make a big mistake if we neglect the clutter. Much worse than using the 
Born approximation (linearization) with respect to p. 


The Random Model 


The clutter (microstructure) is uncertain, so we model it mathematically as a 
random process |1(X), a collection of random variables parametrized by x € R". 
The macrostructure modeled with ¢ and p in (73) is the deterministic part of 
c(X) because it is feasible to recover it from the array data. The wave field 
p(t, X;,Xs) at the array is a random process parameterized by time, the receiver 
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Fig. 6 Plot of the active array data traces for three reflectors in a heterogeneous medium. The 
abscissa is time in «1s and the ordinate is receiver location in A,. The setup of the simulation is 
shown on the /eft where the units in the color bar are Km/s 


and source locations. The data {d(t,X,,X;)} are one realization of this process. If 
we had measurements in many media we could superpose them to approximate 
the statistical average (expectation) E[p(t,X,,X,)] which we call the coherent part 
of the data. This is useful in imaging because we can relate it in a robust way 
to the unknown reflectivity. But in reality there is only one medium, with one 
microstructure given by one realization of the random j1(x). Thus, we do not have 
direct access to the coherent part of the data and must deal with its uncertain, 
incoherent part d(t,X,,Xs) — E[p(t,X,,X;)]. This impedes imaging and causes 
uncertainty in the reconstructions. 

The reflectivity p is deterministic, but the wave field p(t,X,,X;) is random so 
we may view the reconstructions of p in a probabilistic setting. A reconstruction 
is a mathematical model of an imaging function which involves the random field 
P(t, X;, Xs), SO it is a random processes. We can compute only one realization of 
this process because we have only one realization of the random wave field, the data 
{d(t,X,,Xs)}. This realization is what we call an image. 

We study the focusing of imaging functions in random media by looking first 
at E[.7(y°)] and then at its SNR at the peaks. In the case of imaging functions 
like reverse time migration, which are linear in {p(t,xX,,X,)}, the expectation 
of the image is determined by the coherent part of the data E[p(t,x,,x;)]. 
This is not the same as what we would measure in a medium without clut- 
ter. Multiple scattering leads to loss of coherence of the waves, which mani- 
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fests as an exponential decay of E[p(t,X;,X;)] with the distance of propagation. 
The length scale of exponential decay is called the scattering mean-free path 
and it depends on many factors like the wavelength, the range of propaga- 
tion, and the statistics of j(x). Energy is conserved, so the incoherent part of 
the field, the random fluctuations, gain strength. We see in Fig.6 a realization 
of such fluctuations, the many echoes recorded before and after the coherent 
arrivals. 


Time Reversal Is Not Imaging 


We already saw that, superficially, there is a connection between imaging and the 
time reversal process. This is why they are often studied side by side. But the two 
are not the same at all in cluttered media. 

Time reversal works better in clutter [6, 19, 27] because scattering by the 
inhomogeneities spreads out the time reversed wave field captured and then re- 
emitted at the array, as if it came from an array of effective aperture ae > a. The 
more scattering is, the larger the effective aperture, meaning that good focusing 
in cross-range, with resolution of the order A,L/a-, can be achieved with very 
small arrays [19,20]. Moreover, the refocusing is statistically stabile under generic 
assumptions [27]. This is due to the time reversability of the wave equation and the 
fact that the time reversed field propagates to the source in the same medium it came 
from. 

In migration imaging the back-propagation is done analytically or numerically in 
a surrogate medium with wave speed c because we do not know the microscale of 
the real medium. Migration works well only when the clutter effects are weak, as 
shown explicitly in Sect. 7. More precisely, when the array is closer than a scattering 
mean-free path from the imaging region. Otherwise the coherent part of the data, the 
“signal,” is exponentially damped due to scattering in clutter and the incoherent fluc- 
tuations, and the “clutter noise” is significant. The images are difficult to interpret 
and change unpredictably with the realization of clutter. The sole mechanism of 
dealing with the “clutter noise” in migration is the summation over the sensors. If 
we had only additive and uncorrelated noise like in section “Robustness to Additive 
Noise,” it would average out approximately for arrays with many sensors like in the 
law of large numbers. But clutter noise has persistent correlations over the sensors 
and over frequencies, and it cannot be averaged out by summation. More involved 
data processing is needed to mitigate it. 

A natural question arises: Could we improve the migration results if we knew the 
statistics of clutter? Could we just create a realization of the clutter with the given 
statistics and then back-propagate the data in it instead of the medium with speed 
c? The answer is no. In fact we would do much worse than back-propagating in the 
smooth medium because we would essentially double the distance traveled by the 
waves in clutter. The clutter effects are undone only when the waves go back through 
the same medium as is the case in the time reversal process. 
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Fig. 7 System of coordinates and setup for imaging with passive arrays in random media 


5 The Random Travel Time Model of Wave Propagation 


In this section we describe a geometrical optics model of wave propagation in 
random media. It allows an explicit and self-contained analysis of resolution and 
statistical stability of images, and yet it captures canonical scattering effects on the 
wavefront. That is to say, the first two statistical moments of the wave field, which 
describe the loss of coherence and decorrelation of the waves due to scattering in the 
random medium, look the same as those derived from more sophisticated models. 
This makes the random travel time model a good testbed for studying imaging with 
partially coherent waves in random media. 

Consider the setup illustrated in Fig.7, with a passive array at range L from the 
source and the range axis z orthogonal to the aperture A = [-$. 4] x [-$. a We 
use henceforth the notation x, = (x;, L) for the points in the array with cross-range 
components x, € A, and y = (y, 7) for the points in the support of the source, with 
range 7 and cross-range vector y. 

We are interested in the outgoing Green’s function G (w,X,,y) of the Helmholtz 
equation in an medium with random wave speed c(xX) of the form (73) with 
fluctuations around the constant value Cc = c,. The fluctuations are modeled by 
the random process fz assumed stationary and twice differentiable with bounded 
derivatives, almost surely. It has mean zero 


[u(u)] = 0, (74) 


and an integrable covariance 


Ri) =E[wat wna’), (75) 


normalized by 


RO) = 1, i da R(a) = O(1). (76) 
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The length scale € in (73) is the correlation length, the typical size of the 
inhomogeneities, and 0 < 1 models the amplitude of the weak fluctuations of c(x). 
The fluctuations are isotropic, so the correlation length is the same in all directions, 
and the covariance depends on the Euclidian norm of its argument R(u) = R(|ul). 
Any covariance will do, but to simplify the calculation of the statistical moments of 
the Green’s function we take the Gaussian covariance 


R(|u]) =e. (77) 
In the random travel time model the Green’s function is approximated by 


ei? [to (X.Y) +r 5) 


G(o,%,,9) © (78) 


4n|x;, a y| 


with reference travel time t, given by (63) perturbed by the random process v, (X,;, y) 
obtained by integrating ju along the straight ray from y to x,, 


, .  olx,-—yl fs 1-—t)y ¢x, 
Vr(X&,y) = “| an(s 7 Ny + =) ‘ (79) 
0 0 


We refer to [28] for the derivation of the model, which holds in the asymptotic 
regime with separation of scales defined by 


pe ap arcs Op (80) 


and 


o <min 


7 


The high frequency assumption (80) is natural for the geometrical optics approxi- 
mation. Assumption (81) guarantees that ray bending is negligible from the source 
to the array, and the geometrical spreading factor (the amplitude of the Green’s 
function) is approximately the same as in the homogeneous medium. It also gives 
that the travel time t(x,,y), the solution of the eikonal equation in the random 
medium, is given by 


T(X-, ¥) = To(X, Y) + vr(K-, Y) + 4, (82) 
with negligible remainder ét satisfying |w ét| < 1. 
Remark I. Referring to the traces displayed in Fig. 6, the random travel time model 


coupled with the Born approximation describes the echoes from the reflectors, but 
not the many arrivals that occur before and after them. These arrivals are incoherent 
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waves that do not contribute to the resolution analysis of coherent imaging methods, 
as long as they do not dominate the coherent arrivals. Imaging does not work when 
the incoherent echoes dominate unless we filter them out as explained for example 
in [1,2, 10, 13]. 


Long Range Scaling and Gaussian Statistics 


We rewrite (79) as 


eo On) al LE ox 
vr(X;,Y) = On) ONE G3), (83) 
in terms of the random process 
se 1 : y +) 
v(x, y) = ———— | dtu([+—]. (84) 
Gr” =e, . (; £/L 


where £ = |x, —y| andn = ied is the unit vector from the point y in the source to 
the receiver at X,. The normalization by (27)'/* is convenient in the calculation of 
the statistical moments of the Green’s function, for the choice (77) of the covariance 
of the fluctuations. 

We are interested in a long range regime where the waves interact with many 
inhomogeneities as they travel from the source to the array 


£<L=O(L), (85) 


and the random phase in (78) is large. We know from the central limit theorem that 
as L/£ — ov, the right-hand side in (84) converges in distribution to a Gaussian 
process. Thus, the long range assumption (85) allows us to approximate v(x;, y) by 
a Gaussian process, with mean zero and covariance calculated in Appendix 1, 


[o-e) bs 1 _ tf = Sl S 
Re] ~ [ | ar |i Ey ee ~]. 


—oo 20 £L 

(86) 

Moreover, we can estimate the magnitude of the random phase in (78) by 

ee Le 
ove (%, 39) = o(2 i ) (87) 
and ask that it be large 
oV Le 

>i, (88) 
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so that the random medium has a significant effect on the wavefront, and imaging 
becomes difficult. 

When we compare the two assumptions (81) and (88) on o, we see that they are 
consistent if £ >> VLA because then we have 


, HE (2) 
—= < min; —,|— 


VLE L 


i) _ vil 
Big 


Therefore, our regime is defined by the length scale separation 
VAL KLKL, (89) 


and the amplitude o of the fluctuations satisfying 


A VAL 
— seer: (90) 
VLE L 


Statistical Moments 
Since v, is approximately Gaussian, we can estimate the moments of the Green’s 


function in terms of the covariance (86). The calculations simplify for the Gaussian 
covariance (77), where (86) becomes 


3 3) oy > ; Inox) | +090") 4? : _ 1! =x)-FU =i) (y/=y)? 
a [v(x y)v(®. 9) | ~ dte 22 wy dte 22 : 
0 0 


(91) 


This follows by direct integration over ¢ in (86), where (x’ — x), and (y’ — y), 
are the projection of the vectors x — x’ and y’ — y on the plane orthogonal to the 
ray direction n. The second approximation in (91) replaces these projections by the 
cross-range components of x — x’ and y’ — y, and holds under the following scaling 
assumptions on the array aperture and support of the source 


= en —_ 2 
| (x = nl _ |x’ —x| we=0 (=) <1, (92) 
£ £ L , £ £ , 


Loss of Coherence 
Using (91) in the formula 


Come ~ 0 SELES] F (94) 
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which is approximate because v, is only approximately Gaussian, we obtain that 


[6 (o.%.9)] ~ Golo.%, Ne, (95) 


A ao ikL , lula 3 : 
where G,(w,xX;, Y) = aE is the Green’s function in the homogeneous medium and 


4 a) 
S = ———_., k=. 96 
”) J 20 kok Co “a 


This result shows that the coherent part of G, its expectation, is smaller than G;. In 
fact, it is much smaller under our scaling assumption (90), which gives 


L LL 
soi 7 OCG) >» on 
Since 
Nema ttl ie eee 98 
[ (w,x;, y)| | = | o(@,X;,,y)| > (4nL)? , ( ) 


we see that the standard deviation of the field dominates its mean 


2 [I6o.z,.9P] 2 | [6 (.%,.9)]] re |Go(w.%.9)| > | | Gw,%,.3)]| 
(99) 


That is to say, the random phase causes large random fluctuations of G. 
The wave emitted from y and recorded at xX, is given by the Fourier synthesis 


ne ey Ss = 3 
ri = [52 FerGe.%.nem x2 ee 
—oo 270 4nL 

(100) 
It is similar to the wave recorded in the homogeneous medium, but the arrival time of 
the pulse f fluctuates as described by the process v,. When we take the expectation 
of (100) we calculate the envelope of the pulses in (100), and obtain a deformed and 
damped signal centered at time t,(X,, y). Indeed, Eqs. (95) and (100) give that 


292 


oo 2H 4rL 4nL 


’ 


(101) 


> 3 ° d Fw _ 0 io|t—t,(&.¥ t — T)(X;, 
foci] = [CLO goto] - lt oD) 
where we defined the deformed and damped pulse 


fg(t) = i ~ Ce Foye = VE fine, (102) 
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and let 


@? 4cy  w S(o) 


= x , 103 
Vimo £ <° ee 


For example, when the emitted pulse is the Gaussian 


2,2 


- t A V/ 2 _ (@=wo)* 
f(t) = eo i@ot p74 : f(a) = re Be (104) 
the pulse f, takes the form 
f(t) Q ; QO? ; Boe? w 
= ————_ exp} -i | ——— _ ] wt - — ; 
aN? (Gla pt | \ Be 222 +B) 2(02 + B) 


(105) 


Its central frequency is shifted to the lower value 


Q2 
(gpm) <. 


its essential support is wider by a factor of ,/1 + B?/Q2? than that of f(t), and the 
amplitude of the peak is smaller 


05 
fg) = ge << F(0) 1, 


Q 
VO Be 


Remark 2. Formula (95) and the subsequent discussion on pulse deformation 
capture the loss of coherence effects derived from more sophisticated models. The 
loss of coherence manifests as an exponential decay of the mean wave field with 
the distance £ of propagation. The length scale S(w) of decay is the scattering 
mean-free path and it depends on the frequency and the second-order statistics (the 
covariance) of the fluctuations jz. The latter appears in the expression (96) of S as 
the product o7£ in the denominator. 


Statistical Decorrelation of the Waves 
The second moments of the Green’s function are given by 


NIS 


[(ovs (%y)—o! v7 (x" ¥)'| 


1/60. GW'.X.9)| ~ Gy(w,%,9)Go(w',X,¥/e~ 
(106) 
where we used again that v,; is approximately Gaussian. The expectation in the 


exponent follows from definition (83) and the expression (91) of the covariance 
of v, 
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(107) 


The term in the square parenthesis is non-negative, and recalling the scaling relation 
(103), we note that the moments are essentially zero when |x—x’| and/or |y—y’| are 
larger or equal to £. Indeed, the parenthesis in the exponent in the right-hand side of 
(107) is order one in this case and 


etelowas-wnes7] a (ed) <1 


When the cross-range offsets are smaller than the correlation length £, we can 
simplify the expression (107) using the series expansion of the exponential in the 
square bracket, and obtain that 


| G(o,%, 9G (0'.x.9)| xy G4(o,%, 9) Gy (0, X', 9’) (108) 


(oo)? _ |x—x’/ |? +6—x/) + yy) +ly—y/ I? 
e 202 2X2(0) 


Here © = (w + w’)/2 is the central frequency and 


X() = i (109) 


In the derivation of (109) we used that Q < @, as stated by the scaling relation 
(103). The result says that the moments decay exponentially with the cross-range 
offsets, on the scale X(@) « &, which is consistent with the assumption |x — 
x’|, ly — y’'| < @ used in the derivation. For larger cross-range offsets, the moments 
are exponentially small, as noted above. This is precisely what (109) gives, so we 
can use the result for all cross-range offsets, with negligible errors in the analysis of 
imaging in the following sections. 

It is clear that the maximum of (109) occurs at the same frequency w’ = w 
and the same points in the cross-range plane x’ = x and y’ = y. The decay of 
the moments with the frequency and cross-range offsets describes the statistical 
decorrelation of the waves. Indeed, the correlation coefficient is defined by 


Corr [Gx %y), Go’, x’ = 


| G(o.3 %H)G(o',*’ ¥)]- 1[G(o.%.9)| 2 | G(o', 9] 


[var [ée, X, y)| Var [ew x’, v’)] 
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with variance estimated in (99). Recalling from (95) that the mean field is 
exponentially small, we estimate the correlation as 


(w=0!)? _ |x=x! P+ (x=x’) + (y-y’) + ly—y' 7 


(Corr [Gw.% 9), G(o!,%,9)]| we 22 2x2(@) . (110) 


It decays with the frequency offset on the scale Q2, called the decoherence frequency, 
and with the cross-range offsets on the length scale X, called the decorrelation 
length. This scale is proportional to the wavelength, as stated in (109). 


Remark 3. The expression of the moments in (109) is the same as that derived from 
more sophisticated models of cumulative scattering effects in random media. The 
detailed expressions of the decoherence frequency and decorrelation length change 
with the model, but Eq. (109) captures the canonical form of the second moments, 
and thus of the statistical decorrelation of the waves. 


6 Setup for Imaging 


We consider the setup for imaging with passive arrays illustrated in Fig. 7, with the 
range axis originating from the center of the source and the array of aperture A 
orthogonal to it. For active arrays, the excitation comes from a source in A and p is 
the reflectivity that we wish to estimate. In either case the array is assumed planar 
and square, with side length a satisfying the scaling relation (92). We let a be larger 
or similar to the correlation length, so that the Fresnel number ®, satisfies 


a 02 
G2 SL, 111 
iE ar > (111) 


The waves are approximately planar in regimes with smaller Fresnel number and 
we cannot estimate the cross-range location of the support of p. 
We restrict the analysis to a paraxial regime, where we can write 


(112) 


—— x,/2 x,- 


2L L 

4 eae 4 : : 
This holds for apertures satisfying ;7; < 1, and supports of p of radius R in 
cross-range and R,, in range, satisfying ie <«< land R, < aes Merging all our 
assumptions on a, we obtain that 


VAL Ka {min {veL, (aL)! = (aL), (113) 


where the last equality holds in our regime defined by (89) because 
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The summary of the assumptions on the cross-range support of ¢ is 
. LL 
R<« min; VAL, —; = vAL, (114) 
a 


where we recalled (93) and used that 


LL LoL 
= > 1. 
avAL VAL a 
The range support of p satisfies 
TPA 
R, < min |, >} ; (115) 
a 


We take the complex-valued Gaussian pulse (104) for the sake of explicit 
calculations. Its bandwidth B satisfies 


b> 
a KB<Kw. (116) 


The first inequality ensures that f(t) is a pulse of small temporal support, so we 
can determine the travel times 1,(X,,y) = O(L/c,) from the measurements of 
the acoustic pressure p(t,X,) at the array. The second inequality means that all 
the frequencies in the bandwidth are of the order w, and therefore the wavelengths 
satisfy A = A,. The decoherence frequency and decorrelation length defined in 
Sect. 5 obey the scaling relations 


Q Kw, X(@) & X(@) KL Ka. (117) 


To simplify the analysis of the CINT imaging function in Sect. 8, we assume in 
addition that the decorrelation length satisfies 


X(@) K JSAoL. (118) 


Using the definition (109) of X(@,), we see that this is equivalent to asking that 


VAL 


Q<K Wo 
ar) 


K Wo, (119) 


where the second inequality is implied by (89). 
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We analyze in Sects.7 and 8 the imaging function 7(y*) of two methods: 
Kirchhoff migration and coherent interferometry. We use the same symbol for the 
imaging functions to simplify the notation, and define 7 (y’) at points y* in a search 
domain that contains the support of the source. The size of this domain is chosen, 
so that we can use the paraxial approximation (112) for the travel time 1, (x, y°). 
We use for convenience the continuum aperture approximation, which is based on 
the assumption that the sensors in the array are close together. The approximation 
amounts to replacing sums over the sensors by scaled integrals over the aperture A. 
The scaling factor is N/|A|, with N the number of sensors and |.A| = a? the area 
of the aperture. 

The resolution analysis of the imaging function 7(y°) involves two steps: In 
the first step we study the focusing of its expectation. Since the imaging process 
occurs in one medium, corresponding to one realization of the random process LZ, 
we do not have access to E[,7 (y*)]. But if the imaging were robust, the result would 
look almost the same independent of the realization, so 7(y°) would be close to its 
expectation, which is why we study it. The second step of the resolution analysis 
is the assessment of the statistical stability of 7(y*). It amounts to calculating the 
SNR of the image at its peaks. 

Most of our discussion is about imaging with passive arrays. The extension to 
active arrays is straightforward for our random travel time model combined with the 
Born approximation. 


7 Migration Imaging 


In this section we analyze migration imaging with passive arrays. We consider 
only the Kirchhoff migration imaging function because it is essentially the same 
as reverse time migration in the paraxial regime described in Sect. 6. It is given by 


16)=Sp (to(%,. 9°), X,) © am | 2X o8.H).%) x = (x,L), 


r=1 


(120) 


where p(t, x) is the pressure field modeled by 
pe.) = [ dipG)pt.%3), (121) 
R 


in terms of the spatial source density p(y) and the field p(t,x; y) due to a point 
source at y. The latter is defined in (100), and we rewrite it here using the 
simplification of the Green’s function 


a ei lto& Y) +r &.¥)] i 
G(,X;,y) © ~ 
(0,%,,9) 4xL 4xL 


pik (Lan SE —*1¥) tiove iD (122) 
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that holds under the scaling assumptions (113) and (114). The wave field due to the 
point source is given by 


9° 1 S() suleoGiy-t}tione Gi 
tx Y) & dw el lm &Y)—+iove&Y) ~ 
p(t,xy) / fe ae 


a) er ie, Ce ee 
/ dw Lo) oik(z nt Ize 72) Fi@v,(x,y)—iwt : (123) 
ae 82°L 


The Expectation 


Taking the expectation in (120) and using (121), we obtain that 


IGN = | aHoOELIG:H) (124) 


with mean point spread function 


ee N ey eg Es 
ELT: ¥)] a |, D | p (to(X. 9"), ¥)| - (125) 


The expectation of p is given by (101), and using the approximation (112) of the 
travel time we write 


Bie. ab ae oy lhe (y’—y)-x 
T(X, ¥") — To(X, Y) & — Ei = | (126) 
Co L 
for y° = (y’, n°). Equation (125) becomes 
we 
Ne 28?+22) 
D1 7 (y*s y)| ~ 
4rL/ B2/Q2 +1 
de i[kotnt n)+ felt ns Bln! nt mu . 
/ —e (B?/9?-+1) 202 (B2/22-+1) , (127) 
A |Al 


where ky = @,/C, is the central wave number, and it is not difficult to see that the 
mean point spread function peaks at y = (y, 7), as it should. 
To determine the range resolution, we evaluate (127) at y° = y 


. . N Pare ikon’ =n) B? (5 =n)? 
E[ 7° = (y, n°): 9) wy i ae 5 e (B2/27+1) — 2¢2(B2/92-41) , (128) 
IU 
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and estimate the range resolution in terms of the standard deviation of the Gaussian 
In — nl S SVB? +1. (129) 


The result in the homogeneous medium corresponds to letting Q — oo in 

this equation, in which case the range resolution would be c,/B, as stated in 

section “Imaging in Smooth and Known Media.” In our regime the resolution is 

worse due to the broadening of the pulse f, noted in section “Statistical Moments.” 
The cross-range resolution is estimated from 


2 
= = = Wo ——, : ‘ 2 s 2 
Ne Ue/24p dx iko(yS—y)*x/L _ B*|ko(yS—y)*x/L] 
[Ty = (yn): Y)] © @ BMF 209(62/7-+1 
A4nL VB? + Q? Ja |Al 
___% 
Ne 8?7+97) dx —ikoly’=y)*x/L 


(B2/07-+1) 


2 


e 
4nL/B2/22 +1 Ja lAl 


II 


wf 
Ne %8?+9) ee koa(y* —y)-e; 
AnL./B2/22 + 1 2L(B7/Q2 +1) I - 


j=l 


(130) 


The first approximation holds because B and &2 are much smaller than w,, and the 
third line follows by integration over the square aperture A. We estimate the cross- 
range resolution by the support of the main peak of the sinc function 


AL ( B? 
lo’ - yes (F+1). (131) 


It is worse than that in the homogeneous medium of A,L/a because of the shift 
of the central frequency of the deformed pulse f, (t) to a lower value, as noted in 
section “Statistical Moments.” 

The value of the mean point spread function at the peak is given by 


___ #0 
Ne %8?+9?) 


~ 4nL J B2/Q2 + 1 


=| (9:9) | 


S35 N 
(9: y) = —. 132 
K Aly: y) aL (132) 


It is much smaller than J, (¥; y), the peak of the point spread function in the homo- 
geneous medium. Finally, the mean imaging function (124) equals the unknown 
density p integrated against the blurring kernel (127). 
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The SNR 


To simplify the calculations, we consider only the SNR of the point spread function 
J (y¥*; y) evaluated at the peak y° = y. We have 


eed dx 2 al ag Sy NV2n 
sy) XN — p(t(x,y),X: Vy) & —~-——_ x 
I(:¥) [ae (x,y), x;y) 3m? L BLAIS 4 
a4 (o—wo)” . 2S 
i dwe 282 ahve) (133) 
=i 


where we used Eqs. (100) and (122) and the Gaussian pulse (104). The mean square 
follows from (109) 


TT iaeecay?] x N? 
[|7@:9| | sarap If 


cae (w—w0)? _ (w!—w0)? _ (wo!) __ |x/—x/? 
I doda'e 28? 2B? 222 2X2 (wo) 
—o0o 


and changing variables to 


1) a to! ba / = xx ~ J 
Os , @=o-@ and x= ; K=x-x’, 
2 2 
we obtain 
oy 12 N? _ —iP_ (© dd _G=oo? 
q IT | | ~ rre dxe atten ff —e B2 
[Fe 327 BIAl Se i 
a we 2y2 
—oo 82 L2|Al/2B2/Q2 +1 


In the first line of this equation we integrated over x and used the scaling assumption 
(117) on the decorrelation length X to extend the integral over x to the whole plane. 
The second line follows by integrating the Gaussians in x, @ and @. 

The variance of the point spread function at the peak is derived from (132) and 
(134) 


— — rr N?X? 
Var [ 7(¥:y)] = [l7eH1| - | [IG] | ~ 


~ §rL2|,A| V2B2/Q2 41° 
(135) 
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and the SNR is given by 


; Paes 27/02 1/4 wo 
sNR[IG:9)] = ELZ@y]| 1 : os = nl a 
\/ Var [7(: y)| J2n X (B?/Q? + 1) 

(136) 
where we used that | A| = a”. Thus, we see that the SNR improves when we increase 
the aperture and the bandwidth. We restricted the bandwidth by B < ap, so the 
exponentially small factor plays an important role in the SNR. The definition (109) 
of the decorrelation length gives that 


a aw 

$= (i) 

Xx QE 
and to achieve a large SNR we need 


Bi, Q w? 
ex ‘ 
0” @, | 2B? +2) 


The first factor is small, but the exponential is huge and such large apertures cannot 
be realized in practice. Therefore, the SNR of the Kirchhoff migration function is 
small and that the imaging function is not robust. 


8 CINT Imaging 


The statistical instability of migration imaging is due to the large random phase 
wv,(X;, y) in the Green’s function, so a natural idea for improving the imaging is to 
cancel the random phase by data processing. We could cancel the phase exactly by 
working with the intensities | p(w, X,)| of the measurements, but this is not a good 
idea. To see why consider a point source at y, so that 


B(O.%r) = P(O.% 9) = f(@)G(,%,.¥). (137) 
The intensity is 


If? |F@)P 
(4m|x-, yl)? (40)? 


2. %& DP? = |fO)P7IE,%.H? x 


and it is nearly impossible to estimate the location y from it because it is 
approximately constant across the aperture. We need to keep the deterministic phase, 
the travel time t, (X;, y), in order to estimate y. At the same, we should reduce the 
random phase wv, (X;, y). 

The CINT imaging approach accomplishes this by migrating to y* local cross- 
correlations of the data instead of the data themselves. The local cross-correlations 
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are defined by 


C(t, #:%,%) =f ds2®[Q¢—s)] p(s +7/2,8) B(s—F/2,%" 


-[ 5 ef S ge ® a( =) plo +/2,3) Bo — 5/2) 


e iOt— ioe (138) 


They are calculated around the time ¢, in a time window © of duration 1 /Q deter- 
mined by the decoherence frequency Q with a purpose! To explain the advantage of 
working with the cross-correlations (138), we analyze them in section “Analysis of 
the Cross-Correlations for a Point Source” for the case of a imaging a point source. 
Then we study in sections “Resolution Analysis of the CINT Imaging Function” 
and “CINT Images for Passive Arrays as the Smoothed Wigner Transform” the 
resolution of the CINT imaging method with passive arrays. It forms an image by 
migrating to search points y* in the imaging domain the cross-correlations (138). 
Explicitly, the imaging function is given by 


d@ X; — X; > 0) A ~ > 
Ty’) = ae ie ef Ss ul ve ]@(§) te + 472.89 


rr/=1 


DP (@ — 6/2, %) 


= os ees 
ia ie yt rays ay ] iw[to (x, y)—t (X,/ y)] 


xe : ‘ (139) 


where W is another window function that keeps the cross-range offsets x,—x,/ within 
the distance X(q@). As was the case with the time window above, the support X(@) 
is chosen equal to the decorrelation length with a purpose. Recalling from (117) 
that in our scaling regime X(w) ~ X(q,) and using (138), we can approximate the 
CINT imaging function as 


N 
ay X; — X;/ a Se: S5\ ~ 72 2 25\.2 2 
Ty’) ® ) vf Xr) | ¢ Got. 803 eee ce gle ae ca (140) 
rr/=1 a 


This is just the superposition of the migrated local cross-correlations (138) at nearby 
receivers. The migration amounts to evaluating the cross-correlations at the average 
and differences of the travel times from the receivers to the search point 


To (X,, y’) + To (X,/, y’) 
2 ’ 
i= To (Xr, X,Y") = To(K, ¥) — to(K, Y’)- (141) 


t= To(X;, X;, y’) = 
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The resolution analysis below applies to any window functions, but for the sake 
of explicit and simpler formulas, we consider the Gaussian windows 


Os)=e%, Wase?, (142) 


Analysis of the Cross-Correlations for a Point Source 


The model of the Fourier transform of the measurements is given by (137), with 
G(@, X;, y) approximated by (122). Using a notation similar to (141) for the average 
and differences of the travel times, and the Gaussian pulse (104), we get 


CER) % i “doef dao 2) cdr tieleax H-] 
ead 21 (42 L)? B2 J_oo _ Q. 
x ei @[To KX’ Ht] +i (O+G/2)¥2 KY) —i(O-G/2)ve BJ) (143) 


We calculate first the coherent part of C, its statistical expectation, to show that it 
peaks at times 


t= To(X,, Xr’, ¥) and ¢ = To(X;, Xr’, Y) - 


Then we estimate the SNR at the peak to assess its statistical stability with respect 
to the realizations of the random medium. 
The expectation of (143) follows from (109) 


as 1 _ xox’? a (w=wo)” Ie ewty_7 
B(C(, 2:3, 8)] ~ aa ee / dw ew Bt tel RX. Ni] 
21 (42 L)*B — 
oo % @ wf ay 
/ doo (=) oo Fart age) HORE D1] (144) 
te Q 


where we approximated the decorrelation length by that at the central frequency 
Wo, as Stated in the scaling relation (117). Note that the essential support of @ in 
the second integral is limited to |®| < min{Q, B}, because the pulse has finite 
bandwidth, and the waves decorrelate over frequency offsets that are larger than 
Q. Thus, if we had a window ® with Fourier transform supported over a larger 
frequency interval than Q, it would make no difference in (144). However, we will 
see later that it will result in lower SNR, so we would lose in statistical stability. 
Could we take a smaller frequency support of ® than Q? Such a choice would 
reduce the variance of the fluctuations of C, but it would broaden the peak along 
the ¢ axis. This follows from the evaluation of the @ integral in (144). The optimal 
choice of the frequency support is the decoherence frequency (2, as we take it in 
(144). 
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The Mean 
Using the Gaussian window defined in (142), and integrating over w and @ in (144), 
we obtain the following expression of the mean of the correlations 


Ix—x//? 
Jf 200 e@  2X2(wo) 
(4L)2 /4B2/Q2 + 1 


2 ex 712 2 ayy 2 
= ~~ peo B+[io(&.X’ y)—i] B+ [To (&x" y)—t] 
e i@o [io &X Y) i| 4 (4B2/22+41) : (145) 


(CU, 7:%,X)] ~ 


They are indeed peaked at f = T,(x,x’,y) and? = 7,(x,x’, y). The width of the 
peak in 7 is inherited from the O(1/B) support of the pulse. The width of the peak 
int is O (V 1/B2 + 4/ 07). It is similar to the width of the pulse f(t) when the 


bandwidth satisfies B < Q, but it is much broader than f(t) for larger bandwidths 
B > Q. The peak value of the mean is 


max E, [C(t,7:%,¥)] = E[C(@,&,%, 9), 7%, ¥, 9); %, 8) 
_ Ixex’? 
rm /271 e 2X2 (wo) 
~ (nL)? /4B2/Q2 +.1- 
It decays exponentially with the cross-range offset, which is why we restrict it with 


the window W in the CINT function (140). This is essential for improving the SNR 
of the imaging function, as we show next. 


(146) 


The SNR 

We assess the statistical stability of the peak of the cross-correlations by estimating 
the SNR, which is the ratio of (146) and the standard deviation of C(To, 7); X, x’). 
We need the second moment 


r[[CCeo& x x’, Y), T(X, X,Y); XX ®)/"] 


(@1 —@0)~ Stil — ee voy" 
-ooalf da dare B? eel da, da 


Q, Q, 


(147) 


where we let the support of the windows be Q,, not necessarily the same as Q, in 
order to demonstrate how it affects the SNR. The peak value of the mean for this 
choice is a slight modification of (146) 


Imaging in Random Media 1323 


Ix—x’ |? 
J 20 e. 2X2 (wo) 
(4x L)? /2B2/22 + 2B2/22 +1 
(148) 


BC (TX, X.Y), To(X, XY) XX) ] © 


(KX, 9):%,%)) 


and the second moment follows from the calculation in Appendix 2 
8aL4 
1 


= 12 |x—x! [2 ~2 
1 face 202 +1 x 
= 2a-wP2\ 2B? 7 
fate ~ 2 pb @ +1 . (149) 


We distinguish two regimes: 


Nie 


a 


. When |x—x’| >> X(@,) the mean (148) and therefore the SNR are exponentially 
small. The cross-correlations have large random fluctuations in this regime and 
in CINT we make sure that we do not use them by restricting |x — x’| < X(@,) 
with the window function W. 

2. When |x — x’| < X(w,), we can expand the exponential in (149) in series of 

|x —x’|/£ < | to obtain 


2 B2 1 12 Ix—x! [2 RB2 |x — x’ |? 
— (1-]| dte 2 ~— 1, 
@ ( 7. a ) ec Cie 


and simplify (149) as 


zl 


The SNR is bounded by its value at x = x’, in which case Eqs. (148) and (150) 
give 


Gen) 


128734 


(150) 


(8, 9);%, |] 


i, [C@aG, x, y), T(x, a y); X, x)| | Q2 QQ? 


SNR < x Io 5: 
V Var [C (Eo, %. 9), to. ¥.9):%,¥)] >» 78 


(151) 


The subtraction of the random phases in the local cross-correlations (143) at 
nearby receivers reduces substantially the random fluctuations and the SNR is no 


1324 L. Borcea 


longer exponentially small. However, the SNR may still not be large enough and 
it decreases when we increase Q2,, as stated before. A small Q, gives a larger 
SNR but it also reduces the resolution as discussed previously, so Q, ~ Q isa 
good compromise. Equation (151) shows that the stronger the random medium 
effects (i.e., the smaller (2) the lower the SNR. The local cross-correlations are 
less sensitive to the realization of the random medium than the data themselves, 
but still they are not statistically stable. They have random fluctuations that are 
of the same order as the mean. These fluctuations are averaged out in the CINT 
imaging function (140) by the summation over the sensors, if the aperture is large 
enough. This is how we can achieve a statistically stable image, as we show in 
the next section. 


Resolution Analysis of the CINT Imaging Function 


We calculate first the mean of the imaging function to estimate the resolution of the 
expected focusing. The we estimate the SNR and state the conditions under which 
CINT is statistically stable. 


The Mean Point Spread Function 

In the case of a point source at y, the expectation of the imaging function follows 
directly from (140) and (145). Using the Gaussian window W defined in (142) and 
the approximation (126) of the differences of travel times, we obtain 


i) ee ares ma V2aN* 
AG :y)| on a ee 


Ix—x’ |? Lik (yS—y)*(x=x’) Rp? [ofomees B2 [n° n4 (yS—y) , ()f 
dxdx'e **(o) ° 405 B(4B2/92-+1) 7 ae 


xe 
(152) 


Here we used the same notation as in the previous section for the point spread 
function 7(y°; y) to emphasize that the source is a point at y. 

It is convenient to change variables in the integral in (152) to centered and 
difference offsets 


x+x’ 
7 


wr 
II 
tal 
| 
tal 


x= 


and since only |x| < X(@.) « £ < a is in the essential support of the Gaussian, we 
can approximate the right hand side in (152) by extending the integral over x to the 
whole R. Moreover, we note that under our scaling assumptions 


B So 2 _ yx B ko So X 0 
(y’ —y):«—-x) 2 ly’ —yl @o) 2 1, 
Co L Wo L 
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so we can neglect the third term in the exponential in (152). This is for search points 
y® satisfying 


ly’ —yl Wo 
jes <<, 
AoL/X(@) ~ B 


where A,L/X(q,) turns out to be the cross-range resolution limit, as shown below. 
We obtain after integrating in x that 


2/N*X? (a) — 4 toXtooply? al)” 


D| 7(y*sy)| * e 
| 16L?|A|./4B2/Q2 +1 
a 2 yx? 
/ 7 Faas? ie ; (153) 
A |Al 


and note that the right-hand side peaks at y* = y, as it should. 


Range Resolution When we evaluate (153) at y° = y, we obtain 


J2/nN?X7(a,) Big 


eo BaBety | (154) 
16L2|A| /4B2/Q2 + 1 


The range resolution is given by 


In’ — nl S © /4B7/ 92+ i, (155) 


ELT = (y.n°); 9) | 


It is worse than in the homogeneous medium because of the factor /4B2/Q? + 1. 
This factor is large when the decoherence frequency satisfies Q <«< B and reduces 
the range resolution to O(c,/Q). 


Cross-Range Resolution Evaluating (153) at 7° = n, we obtain 


DT (¥'s¥)] © /1/2N?X? (wo) oi[eege =f 
32L2|,A|/4B2/Q2 +1 


2 erf 


I] aB Se ae 
j=l ST OT se yi 


Bets yy, 
| tien eae | 
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Both factors are peaked at y° = y, so we estimate the resolution by the width of the 
tighter peak 
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Aol AoL j4w2 7 we 
X(@) a Q2 Be 
This is worse than the resolution A,L./a in homogeneous media because in our 
scaling X(w,) < a and 4/ op oe > 1. 


The SNR 

To calculate the SNR, we need the variance of 7(y*; y) and therefore the fourth- 
order moments of the Green’s function. Their calculation is straightforward, due to 
the Gaussianity of v,, but laborious. We state directly the result which distinguishes 
between two cases: 


ly’ —y| < min (156) 


1. When |Ax| = |x — x’| >> @, the fourth moments 


factorize in the product of second moments 
Ma(o, 0', 0,0, x, x’, %,%) & M2(o, 0, x, 9) M2(o!, 0, x’, X), 


where 


WC G2 = [olor Gris) i(o-4) ve (& 9) i gece aaa. 


and x = (x, L), x= (x, 0). The same notation applies for the prime variables. 
2. When | Ax| < @, we have the expression 


Ma(o, 0, 0, 0',x, x’, XX!) & M2(0, 0, x, 3) M2(0!, 0’, x’, X’) 


ey —Plas? | Oo! 17x +x’ t?|Ax|? 
x exp f te 2 2 Bae gre) 1- 72 


52 Ax ox ox’ 
€ \QX(@,) 2X) ) |{ 


We no longer have the factorization of the moments, and the exponential factor 
is bounded above and below by constants of order one. 
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The factorization of the moments for |Ax| >> & means that only the set {x,x’ € 
A,  |x—x’| < £} contributes to the calculation of the variance of 7 (y*). Explicitly, 
we have 


—_ N4 foe) oe) a 
Var [ 7 (9; ¥)] © aoe | dode’ ff dade ff dxdx'1 (0,4 


laa =o) _(~24 iy. ix12-+1x” [2 
(x—x) ff dxdx' xe (45) )(Sr+ aps) 2K2(w) 


_ Axl? 
1 a2 | Ga! 4.3 PK! Plast (ok ox! 
= je dte 2 | +392 Gp) (1 )-v3r 4 F (aFS5 aficcy) =i 


This is a complicated expression but we can estimate it by replacing the last factor 
by an order one constant. We obtain that 


.., ne 
a[IED)~ IE[7G:y)]] . (157) 


where the symbol ~ means approximate, up to a multiplicative constant of order 
one. The SNR at the peak is given by 


3155 a 
SNR[J(¥;¥)] ~ 5 (158) 
where we used that |.A| = a”. Thus, the CINT point spread function 7(y’; y) has a 
robust (statistically stable) focusing at the location y of the source only if the array 
has aperture a > €. This is a significant improvement over the result in Sect. 7, 
where for the migration method to be stable we needed that a/£ be much larger 


than a huge number, not one like above. 


Remark 4. Our analysis above is restricted to the CINT point spread function. It 
extends easily to the case of a distributed source density. We do not do it here 
because the calculations are long and the result does not bring any significant insight 
to the problem. 


CINT Images for Passive Arrays as the Smoothed Wigner Transform 


The Wigner transform, also called the Wigner distribution, is a classic tool for study- 
ing high frequency limits of the wave equation. It is very useful in imaging because 
it allows us to extract the important phase information from the measurements at the 
array: the travel times and the direction of arrival of the waves from the unknown 
source location. 
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The Wigner transform of the wave field p(w, (x, L)) evaluated in the plane of 
the array, at range L from the source, is defined by 


dx do . x 
W(a,x;t, K) = om! 5 i —pilo+— 5" 2 (x+ 5+b) 


We cannot compute it in practice because the measurements of p are limited to the 
aperture, but we shall see in this section how W is related to the CINT imaging 
function. Note that W is real valued and its integral over the last two arguments 
gives the intensity of the wave field 


aK" [ dt W(w,x:t, K) = |p (a, (x, L))|’. (160) 
R2 —00 


Thus, we may think of W as the “energy density” of the waves resolved over the 
phase space variables ¢ and K, although W is not positive in general. The variable 
t is dual to the frequency offset @, so it has units of time. The variable K is dual to 
the cross-range offset, and is called the slowness vector because its units are time 
over length, like 1/co. 

We use the paraxial approximation (112) of the travel time and assumption (118) 
to rewrite the expression (139) of the CINT imaging function 7(y*) evaluated at 


y’ = (y’, 7°) as 


neve: camel. iw [ 400(5) [arf «oa 
«p(o+S.(x+5.2)) F(o-F. (x-5.2)) 


_ cg -w (x—y5)*® 
eit (&L)F jie SS ; (161) 


The cross-range variables x and x should take values in the set defined by the 
constraint x + x/2 € A, so that the wave field is evaluated in the aperture of the 
array. Nevertheless, since the window W restricts |x| to the decorrelation length 
X(@,), which is much smaller than the aperture a, we can approximate 7(y°) by 
extending the integral over x to the whole plane and letting x vary in A. 

The relation between 7(y*) and W follows from (161), after inverting the 
transform in (159), 


20 y2 
T(¥) & a dof dxf af dK w’W(a,x:t, K) 


x &[Q(r,((x, L), ¥°) —t)] [on Xe) (x-*=")]. (162) 
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Thus 7(y*) is given by the Wigner transform smoothed over all its arguments. 
The smoothing is by the integration over the bandwidth and the aperture and by 
the convolution with the windows. The time window ® has support of order 1/Q, 
and the convolution is evaluated at the travel time from the search point y* to the 
receiver location X = (x, L) in the array. The window W over the slowness vectors 
has support of order @,X(@)) = /3Q£, and the convolution is evaluated at the 
slowness vector (x — y*)/(coL), which is approximately the cross-range gradient of 
the travel time Vx» (x, y’). 


Remark 5. It turns out that the Wigner transform is weakly self-averaging for a 
wide range of wave scattering regimes in random media. That is to say, although 
W (a,x; t, K) is random when evaluated pointwise, it becomes deterministic when 
integrated against some test function y(w,x;t,K), as in (162). The weak self- 
averaging property of the Wigner transform is the basis of the proofs of statistical 
stability of the time reversal process [3,27] and of CINT imaging in random media 
[9]. It is an asymptotic result obtained by taking various limits, depending on the 
scaling regime. The smoothed Wigner transform is not deterministic in practice 
because the limit is never realized, but its random fluctuations at the peaks are small, 
i.e., the SNR is large. 


Calculation of the Wigner Transform 
Here we calculate the Wigner transform in order to show explicitly the role of the 
smoothing by the windows ® and W used in the local cross-correlations of the array 
data. Again, we assume for simplicity a point source at y. 

Using the model (137) of the array data with Green’s function (122) in the 
definition (159) of the Wigner transform, we obtain 


1 ~ [ies O\ > Oo 
W(a,x;t, K) acaba [ttf dof (ot S)F (o-$) 
x oll ( (4 $1) §)—c((x-§.4)3)-K-5] +79 [e( (e+ §.2) 5) +c((x-§.2) 3) 2] 


Here t is the random travel time 
T(X, Y) = To(X, Y) + V(X, y), 


and assuming that the pulse 7 is modeled by the Gaussian (105), we can integrate 
over @ in (163) 


e7(@-@)"/B? 


(nL Se 
x Olt ( (5+ 5.4) 5)—e( (x 3.2) 3) -K-5] (164) 


W(a,x;t, K) & dx oF [e( (e+ $1) 5) +e((x- 9.2) 3) 21] 
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It remains to evaluate the x integral which we cannot do explicitly in (164) because 
there is no restriction on the cross-range offset x. However, when we smooth W 
over the slowness vectors, as we do in the convolution with the window W in (164), 
we see that only small cross-range offsets contribute to the results. Explicitly, we 
have 


2./me (eo) / BY 
(4 L)? Bla, X (wo) 


,) givo| Var( (x2) 3)—K]-% (165) 


/ dK'W [w, X(w,)(K’ — K)] W(@, x; 1; K’) © 
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erie af ax ( 
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where we used our scaling assumptions to approximate 
B Xx as x me 7 
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for cross-range offsets in the support of W, satisfying |k| < X(@,). Now we can 
integrate over x in (165) and over w 


/ dK'v [w, X (wo) (K’ — Ww] f- dw W(a,x;t; K’) 
R¥ —oo 


20 —B2[r((x,L)9)-t] 5 G > 
* yar? HOOT x h [wo X(w) (Vet ((x, L).¥) —K)]. 166) 
We conclude from (166) and the paraxial approximation (112) of the determin- 
istic travel time t, that the Wigner transform smoothed over the slowness vector 
peaks at the random arguments 


= = +Vxv-((x, L),9). (167) 


t = t ((x,L),y) +, ((%,L),y) and K= 


Co 


The peak dances around the point (to((x, L), y). <3) in the (t, K) phase space, as 
modeled by v, in the ¢ direction and by Vxv; in the plane of the slowness vectors K. 
In the CINT imaging function (163) we evaluate (166) at the expected peak location 
(« ((x, L), y’), —) supposing that the source is at y°. The role of the window Ny 
is to mitigate the peak dancing in K. The standard deviation of the location of the 
peaks is quantified in our model as O(1/(@X(@.))), which is precisely the support 
of W in (166). Thus, in CINT we are essentially taking the envelope of the random 
peaks in K to stabilize statistically the image, and thus increase the SNR. 

The standard deviation of the peak location in ¢ is O(1/&) in our model, and 
the peak dancing is significant in (166) when B >> Q. The CINT imaging function 
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(163) is statistically stable because of the convolution with the window © of support 


O(1/2). 


Remark 6. The results above reveal that on one hand the windows used in the calcu- 
lation of the local cross-correlations are needed to stabilize the imaging process, but 
on the other hand they blur the peaks of W and therefore of the image. In practice 
it is unlikely that we would know the statistics of the medium fluctuations so we 
cannot calculate the decorrelation length X(@,) and the decoherence frequency Q 
using the formulas (103) and (109). In fact these formulas may not even apply 
because the random travel time model may not be a good approximation of the 
net scattering effects of the medium. It is the form of the second moment formulas 
(109) which is general, and not the detailed definition of Q and X(q@,) that the CINT 
imaging methodology should use. The scales X(@,) and Q should not be set based 
on a specific model, they should be estimated as part of the imaging process. This 
can be done in principle directly from the data, by calculating for example empirical 
correlations and estimating how they decay with the frequency and cross-range 
offsets. Alternatively, we may estimate X(w,) and Q while we form the image, 
by exploiting the trade-off between the resolution (sharpness) of the image and its 
SNR. The latter approach is known as adaptive CINT. It uses a figure of merit of the 
quality of the image and estimates X(@,) and Q by optimizing it during the image 
formation [8]. 


CINT Imaging with Active Arrays 
The CINT imaging function for active arrays back-propagates the local cross- 


correlations of the array measurements p(w, X;,X,;) to the search points y*, using 
the roundtrip travel times 


To(Xr, ¥°, Xs) = To(X, ¥°) + To(Xs, Y’)- (168) 


Here x, = (x,, L) and x, = (x,, L) denote the location of the receivers and sources 
in the aperture A, and the imaging function is given by 
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(169) 


The resolution analysis of (169) is more involved than that for passive arrays, even 
for the Born model of the data 
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because it requires higher moments of the random travel time perturbation v,. We 
do not include it here because it does not bring any new insight. We only highlight 
the relation between the imaging function and the Wigner transform, in the case of 


a point-like reflector at y. 
Using the model 


B(@, &.%s) © f (@)G(@, %,. 9) G(w, Xs. ), 
we can rewrite (169) in terms of the Wigner transform W(@,x;t,K) of 


f (w)!/ Gia, X, y), which is like the square root of the data. Explicitly, we have 
that 
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where we let 


= X; + X;/ ee 
xX, = —— and x, =x,—-¥X,, 
2 


integrated over @, and used the paraxial approximation (112) of the travel time and 
the assumption (118) on the decorrelation length. 

The result (171) is similar to that for passive arrays. To see this more explicitly, let 
us suppose that NV, = N; and use the continuum aperture approximation to rewrite 
(171) as 
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The K and K’ integrals are just like (165), except that now we have the square root 
of the pulse, so we can write directly the result from there, 


(09) 
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We see the same random peak dancing as before, which is mitigated in the slowness 
plane by the window W of width chosen appropriately as 1/(@,X(w,)). The peak 
dancing in ¢ is visible in the case B >> Q, but it is mitigated in the imaging function 
by the convolution with the window ®. This can be seen from (172), once we use 
the Gaussian window (142) and integrate in ¢ and t’ 


N*X4(a,)(2B?/Q? + 1)7/? 


as J2n(40L)4| Alt 
| dxdxG| 0, X(0») (Yrs, L),y) -— *)I 
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Numerical Simulations 

We illustrate with numerical results the point spread function of CINT with active 
arrays. The array data p(w,X,,X;) is simulated with the random travel time 
approximation (78) of the Green’s function in a medium with wave speed modeled 
by (73) for a constant € = cy. The standard deviation of the fluctuations of c(X) is 
eithero = 0.04 % oro = 0.1 %, and the correlation length is 2 = 100A,. The array 
has aperture a = 4€. It is centered at the cross-range location y of the reflector, 
and at range L = 99£ from it. There are N = 101 sensors in each direction. All 
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Fig. 8 Traces p(t, X,, X,) in the homogeneous medium (fop) and in one realization of the random 
medium, for 0 = 0.1%. The traces are similar, except for the wavefront distortion noted in the 
bottom plot 


of them are receivers except for the central one, which is also a source. It emits a 
pulse f(t) that is a modulated sinc function, with constant Fourier coefficients in 
the bandwidth [125, 175] kHz. The central wavelength is A, = 2cm. 

We display in Fig.8 the simulated data (traces) p(t,X,,X;) at the receiver 
locations along one cross-range line in the array, with the source in the middle. 
Each trace is obtained by Fourier synthesis of p(w, xX,,X;), for a discretization of 
the bandwidth in one hundred equidistant frequency intervals of 0.5 kHz. The top 
plot in Fig. 8 is for the homogeneous medium and the bottom plot for one realization 
of the random medium, with o = 0.1 %. The traces look similar, but we note in the 
bottom plot that the wavefront is distorted. 

We show in Fig.9 the sample estimates of the mean Kirchhoff migration and 
CINT imaging functions, normalized by the standard deviation at their peak, which 
is at the reflector location y. We display the square root for the CINT image, so we 
can compare the resolution of the two methods. If we did not have the windows 
in CINT, the imaging function would be the square of the migration one, which is 
why we take the square root in Fig. 9. We note that, as predicted by the theory, the 
CINT image is blurier than the migration one, but it has a much higher SNR at y. 
Kirchhoff migration gives a very small SNR, so the images are expected to change 
a lot with the realizations of the random medium. The SNR of the CINT image is 
of order one in this simulation because the aperture a is not very large. Thus, the 
CINT images change with the realizations as well, but the changes are smaller and 
never exceed the mean. The peak is expected to stay close to y, independent of the 
realization. This is illustrated in Fig. 10 where we display the migration images (top 
row) and CINT images (bottom row) in three realizations of the random medium. 
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Fig. 9 The mean of the Kirchhoff migration (left) and square root of CINT images (right) divided 
by their standard deviation at the peak. The colorbar shows that the SNR of CINT is approximately 
100 times larger than that of Kirchhoff migration. The axes are range and cross-range in Aj, 
measured from the source at the array. The true reflector location is in the middle of the search 
domain, and it is indicated with the white circle 


The migration images are quite different from each other: the first image is not as 
well focused as the others and the peak changes its location. The CINT images 
change much less with the realization and are very similar to their mean displayed 
in the right plot in Fig. 9. 


9 Appendix 1: Second Moments of the Random Travel Time 


We obtain by direct calculation from (75) and (84) that 
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where we made the change of variables (t, t’) ~ (f, t’) with 
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We can rewrite the result as 


1 lo.) dt 7 
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ae ay ii _2 ' : 
and note that the integrand converges to R [i n+ wees] pointwise, 
from below, as £/£ — oo. Since R is integrable by assumption, we obtain from the 


Lebesgue dominated convergence theorem that 
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as stated in (86). 


Appendix 2: Second Moments of the Local 
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Cross-Correlations 


Using the Gaussian windows in (147), we obtain 
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and it remains to evaluate the expectation. Because v, is approximately Gaussian, 


we have 
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where we let Aw = @ — @2 and A® = @) — @. The exponent follows from (84) 
and (91) 
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Substituting in (177), we obtain 
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and after evaluating the Gaussian integrals, we obtain (149). 


11 Conclusion 


This article reviews basic results on coherent array imaging in random media. The 
random model is motivated by the uncertainty of the small-scale fluctuations of 
the wave speed in complex media with numerous inhomogeneities. We consider 
a simple model of wave propagation in random media that captures nevertheless 
canonical scattering effects on the coherent part of the waves, and consider two 
imaging methods: migration imaging and CINT imaging. They are both related to 
an approximation of the solution of the least squares data fit formulation of the 
inverse problem. Migration imaging is superficially connected to the time reversal 
process, in the sense that it involves the back-propagation to the imaging region of 
the time reversed waves measured at the receivers in the array. However, the back- 
propagation is in a surrogate medium, not in the real one as in the time reversal 
process, because the medium is not known in imaging. We know only its smooth 
part, but not its inhomogeneities, which is why we model it as random. This subtle 
difference between imaging and time reversal has profound effects in random media. 
We give an explicit and self-contained study of these effects and show that migration 
imaging is not useful when the waves propagate longer than a scattering mean- 
free path in random media. The CINT method images by back-propagating to the 
imaging region local cross-correlations of the measurements at the array. We analyze 
in detail these local cross-correlations in order to explain why they are useful in 
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imaging. Moreover, we give an explicit resolution analysis of CINT, which includes 
an assessment of its statistical stability, and illustrate the results with numerical 
simulations. 
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Abstract 

The theme of this chapter is statistical methods in imaging, with a marked 
emphasis on the Bayesian perspective. The application of statistical notions and 
techniques in imaging requires that images and the available data are redefined in 
terms of random variables, the genesis and interpretation of randomness playing a 
major role in deciding whether the approach will be along frequentist or Bayesian 
guidelines. The discussion on image formation from indirect information, which 
may come from non-imaging modalities, is coupled with an overview of how 
statistics can be used to overcome the hurdles posed by the inherent ill-posedness 
of the problem. The statistical counterpart to classical inverse problems and 
regularization approaches to contain the potentially disastrous effects of ill- 
posedness is the extraction and implementation of complementary information 
in imaging algorithms. The difficulty in expressing quantitative and uncertain 
notions about the imaging problem at hand in qualitative terms, which is a 
major challenge in a deterministic context, can be more easily overcome once 
the problem is expressed in probabilistic terms. An outline of how to translate 
some typical qualitative traits into a format which can be utilized by statistical 
imaging algorithms is presented. In line with the Bayesian paradigm favored 
in this chapter, basic principles for the construction of priors and likelihoods 
are presented, together with a discussion of numerous computational statistics 
algorithms, including maximum likelihood estimators, maximum a posteriori and 
conditional mean estimators, expectation maximization, Markov chain Monte 
Carlo, and hierarchical Bayesian models. Rather than aiming to be a comprehen- 
sive survey, the present chapter hopes to convey a wide and opinionated overview 
of statistical methods in imaging. 


1 Introduction 


Images, alone or in sequences, provide a very immediate and effective way of 
transferring information, as the human eye—brain complex is extremely well adapted 
at extracting quickly their salient features, let them be edges, textures, anomalies, or 
movement. While the amount of information that can be compressed in an image is 
tremendously large and varied, the image processing ability of the human eye is so 
advanced to outperform the most advanced of algorithms. One of the reasons why 
the popularity of statistical tools in imaging continues to grow is the flexibility that 
this modality offers when it comes to utilizing qualitative attributes of the images or 
to recover them from indirect, corrupt specimens. The utilization of qualitative clues 
to augment scarce data is akin to the process followed by the eye—brain system. 
Statistics, which according to Pierre-Simon Laplace, is “common sense 
expressed in terms of numbers,” is well suited for quantifying qualitative attributes. 
The opportunity to augment poor quality data with complementary information 
which may be based on our preconception of what we are looking for or on 
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information coming from sources other than the data makes statistical methods 
particularly attractive in imaging applications. 

In this chapter, we present a brief overview of some of the key concepts and 
most popular algorithms in statistical imaging, highlighting the similarity and the 
differences with the closest deterministic counterparts. A particular effort is made 
to demonstrate that the statistical methods lead to new ideas and algorithms that the 
deterministic methods do not give. 


2 Background 
Images in the Statistical Setting 


The mathematical vessel that we will use here to describe a black and white image 
is a matrix with nonnegative entries, each representing the light intensity at one 
pixel of the discretized image. Color images can be thought of as the result of 
superimposing a few color intensity matrices; in most application, a color image 
is represented by three matrices, for example, encoding the red, green, and blue 
intensity at each pixel. While color imaging applications can also be approached 
with statistical methods, here we will only consider gray-scale images. Thus, an 
image X is represented as a matrix 


X= [sy], Issn lsej am, x20. 


In our treatment, we will not worry about the range of the image pixel values, 
assuming that, if necessary, the values are appropriately normalized. Notice that this 
representation tacitly assumes that we restrict our discussion to rectangular images 
discretized into rectangular arrays of pixels. This hypothesis is neither necessary 
nor fully justified, but it simplifies the notation in the remainder of the chapter. In 
most imaging algorithms, the first step consists of storing the image into a vector by 
reshaping the rectangular matrix. We use here a columnwise stacking, writing 


X= [xO xO eM], xD eR" Ls j<m, 


and further 


x) 


x = vec (X) = : eRX, N=nxm. 


x) 


Images can be either directly observed or represent a function of interest, as is, 
for example, the case for tomographic images. 
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Randomness, Distributions, and Lack of Information 


We start this section by introducing some notations. A multivariate random variable 
X : Q —> RY is a measurable mapping from a probability space Q equipped 
with a o-algebra and a probability measure P. The elements of R%, as well as 
the realizations of X, are denoted by lowercase letters, that is, for @ € Q given, 
X(w) = x € R™. The probability distribution jvy is the measure defined as 


ux(B) = P(X7'(B)), BCR" measurable. 


If zy is absolutely continuous with respect to the Lebesgue measure, there is a 
measurable function zy, the Radon—Nikodym derivative of 1x with respect to the 
Lebesgue measure such that 


px(B) = f mx(xds. 


For the sake of simplicity, we shall assume that all the random variables 
define probability distributions which are absolutely continuous with respect to the 
Lebesgue measure. 

Consider two random variables X : Q > R% and ¥Y : Q > R™. The joint 
probability density is defined first over Cartesian products, 


pxy(B x D) =P(X—(B)NY~(D)), 


and then extended to the whole product o-algebra over RY x R™. Under the 
assumption of absolute continuity, the joint density can be written as 


pxv(Bx D)= ff mvy(x »pdydx, 
BJD 


where zy,y is a measurable function. This definition extends naturally to the case 
of more than two random variables. 

Since the notation just introduced here gets quickly rather cumbersome, we will 
simplify it by dropping the subscripts, writing zy y (x, y) = (x, y), that is, letting 
x and y be at the same time variables and indicators of their parent uppercase 
random variables. Furthermore, since the ordering of the random variables is 
irrelevant — indeed, P (X~'(B)  Y~!(D)) = P(Y7!(D)N X7!(B)) — we will 
occasionally interchange the roles of x and y in the densities, without assuming that 
the probability densities should be symmetric in x and y. In other words, we will 
use zr as a generic symbol for “probability density.” 

With these notations, given two random variables X and Y, define the marginal 
densities 


mx) =f meyddy 0) = fm. yids. 


Statistical Methods in Imaging 1347 


which express the probability densities of X and Y, respectively, on their own, while 
the other variable is allowed to take on any value. By fixing y, and assuming that 


(vy) 4 0, we have that 
[Dera 
RN 7(y) 


hence, the nonnegative function 


(1) 


defines a probability distribution for X referred to as the conditional density of X, 
given Y = y. Similarly, we define the conditional density of Y given X = x as 


def I(X, Y) 
u(y | x)= x(x) 


(2) 


This rather expedite way of defining the conditional densities does not fully explain 
why this interpretation is legitimate; a more rigorous explanation can be found in 
textbooks on probability theory [8, 18]. 

The concept of probability measure does not require any further interpretation 
to yield a meaningful framework for analysis, and this indeed is the viewpoint of 
theoretical probability. When applied to real-world problems, however, an interpre- 
tation is necessary, and this is exactly where the opinions of statisticians start to 
diverge. In frequentist statistics, the probability of an event is its asymptotic relative 
frequency of occurrence as the number of repeated experiments tend to infinity, 
and the probability density can be thought of as a limit of histograms. A different 
interpretation is based on the concept of information. If the value of a quantity is 
either known or it is at potentially retrievable from the available information, there 
is no need to leave the deterministic realm. If, on the other hand, the value of a 
quantity is uncertain in the sense that the available information is insufficient to 
determine it, to view it as a random variable appears natural. In this interpretation 
of randomness, it is immaterial whether the lack of information is contingent 
(“imperfect measurement device, insufficient sampling of data’) or fundamental 
(“quantum physical description of an observable’). It should also be noted that the 
information, and therefore the concept of probability, is subjective, as the value of 
a quantity may be known to one observer and unknown to another [14, 18]. Only in 
the latter case the concept of probability is needed. The interpretation of probability 
in this chapter follows mostly the subjective, or Bayesian tradition, although most 
of the time the distinction is immaterial. Connections to non-Bayesian statistics are 
made along the discussion. 

Most imaging problems can be recast in the form of a statistical inference 
problem. Classically, inverse problems are stated as follows: Given an observation 
of a vector y € R™, find an estimate of the vector x € R, based on the forward 
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model mapping x to y. Statistical inference, on the other hand, is concerned with 
identifying a probability distribution that the observed data is presumably drawn 
from. In the frequentist statistics, the observation y is seen as a realization of a 
random variable Y , the unknown x being a deterministic parameter that determines 
the underlying distribution z(y | x), or likelihood density, and hence the estimation 
of x is the object of interest. In contrast, in the Bayesian setting, both variables x and 
y are first extended to random variables, Y and X, respectively, as discussed in more 
detail in the following sections. The marginal density z(x), which is independent 
of the observation y, is called the prior density and denoted by Zprior(x), while the 
likelihood is the conditional density z(y | x). Combining the formulas (1) and (2), 
we obtain 


Tprior (x) (y | x) 


m(x|y)= oh 


’ 


which is the celebrated Bayes’ formula [3]. The conditional distribution (x | y) is 
the posterior distribution and, in the Bayesian statistical framework, the solution of 
the inverse problem. 


Imaging Problems 


A substantial body of classical imaging literature is devoted to problems where the 
data consists of an image, represented here as a vector y € R™ that is either a noisy, 
blurred, or otherwise corrupt version of the image x € R™ of primary interest. The 
canonical model for this class of imaging problems is 


y = Ax + “noise,” (3) 


where the properties of the matrix A depend on the imaging problem at hand. A 
more general imaging problems is of the form 


y = F(x) + “noise,” (4) 


where the function F : RY t+ R™ may be a nonlinear function and the data 
y need not even represent an image. This is a common setup in medical imaging 
applications with a nonlinear forward model. 

In classical, nonstatistical framework, imaging problems, and more generally, 
inverse problems, are often, somewhat arbitrarily, classified as being linear or 
nonlinear, depending on whether the forward model F in (4) is linear or nonlinear. 
In the statistical framework, this classification is rather irrelevant. Since probability 
densities depend not only on the forward map but also on the noise and, in the 
Bayesian case, the prior models, even a linear forward map can result in a nonlinear 
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estimation problem. We review some widely studied imaging problems to highlight 
this point. 


1. Denoising: Denoising refers to the problem of removing noise from an image 
which is otherwise deemed to be a satisfactory representation of the information. 
The model for denoising can be identified with (3), with M = N and the identity 
A=1e€R%*" as forward map. 

2. Deblurring: Deblurring is the process of removing a blur, due, for example, 
to an imaging device being out of focus, to motion of the object during 
imaging (“motion blur’), or to optical disturbances in atmosphere during image 
formation. Since blurred images are often contaminated by exogenous noise, 
denoising is an integral part of the deblurring process. Given the image matrix 
X = [x;;], the blurring is usually represented as 


y= Y aijxexne + “noise.” 
ke 


Often, but not without loss of generality, the blurring matrix can be assumed to 
be a convolution kernel, 


Gij,.ke = Gi-k,j—€, 


with the obvious abuse of notations. It is a straightforward matter to arrange the 
elements, so that the above problem takes on the familiar matrix—vector form 
y = Ax, and in the presence of noise, the model coincides with (3). 

3. Inpainting: Here, it is assumed that part of the image x is missing due to an 
occlusion, a scratch, or other damages. The problem is to paint in the occlusion 
based on the visible part of the image. In this case, the matrix A in the linear 
model (3) is a sampling matrix, picking only those pixels of x € R% that are 
present in y ¢ RY, M <N. 

4. Image formation: Image formation is the process of translating data into the form 
of an image. The process is common in medical imaging, and the description of 
the forward model connecting the sought image to data may involve linear or 
nonlinear transformations. An example of a linear model arises in tomography: 
The image is explored one line at the time, in the sense that the data consist of line 
integrals indirectly measuring the amount of radiation absorbed in the trajectory 
from source to detector or the number of photons emitted at locations along the 
trajectory between pairs of detectors. The problem is of the form (3). An example 
of a nonlinear imaging model (4) arises in near-infrared optical tomography, in 
which the object of interest is illuminated by near-infrared light sources, and the 
transmitted and scattered light intensity is measured in order to form an image of 
the interior optical properties of the body. 
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Some of these examples will be worked out in more details below. 


3 Mathematical Modeling and Analysis 
Prior Information, Noise Models, and Beyond 


The goal in Bayesian statistical methods in imaging is to identify and explore 
probability distributions of images rather than looking for single images, while 
in the non-Bayesian framework, one seeks to infer on deterministic parameter 
vectors defining the distribution that the observations are drawn from. The main 
player in non-Bayesian statistics is the likelihood function, in the notation of 
section “Randomness, Distributions and Lack of Information,” z(y | x), where 
Y = Yobserved- In Bayesian statistics, the focus is on the posterior density (x | y), 
Y = observed, the likelihood function being a part of it as indicated by Bayes’ 
formula. 

We start the discussion with the Bayesian concept of prior distribution, the non- 
Bayesian modeling paradigm being discussed in connection with the likelihood 
function. 


Accumulation of Information and Priors 


To the question, what should be in a prior for an imaging problem, the best answer 
is whatever can be built using available information about the image which can 
supplement the measured data. The information to be accounted by the prior can be 
gathered in many different ways. Any visually relevant characteristic of the sought 
image is suitable for a prior, including but not limited to texture, light intensity, 
and boundary structure. Although it is often emphasized that in a strict Bayesian 
framework the prior and the likelihood must be constructed separately, in several 
imaging problems, the setup may be impractical, and the prior and likelihood need to 
be set up simultaneously. This is the case, for example, when the noise is correlated 
with the signal itself. Furthermore, some algorithms may contain intermediate steps 
that formally amount to updating of the a priori belief, a procedure that may 
seem dubious in the traditional formal Bayesian setting but can be justified in the 
framework of hierarchical models. For example, in the restoration of images with 
sharp contrasts from severely blurred, noisy copies, an initially very vague location 
of the gray-scale discontinuities can be made more precise by extrapolation from 
intermediate restorations, leading to a Bayesian learning model. 

It is important to understand that in imaging, the use of complementary infor- 
mation to improve the performance of the algorithms at hand is a very natural 
and widespread practice and often necessary to link the solution of the under- 
lying mathematical problem to the actual imaging application. There are several 
constituents of an image that are routinely handled under the guidance of a priori 
belief even in fully deterministic settings. A classical example is the assignment of 
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boundary conditions for an image, a problem which has received a lot of attention 
over the span of a couple of decades (see, e.g., [21] and references therein). In 
fact, since it is certainly difficult to select the most appropriate boundary condition 
for a blurred image, ultimately the choice is based on a combination of a priori 
belief and algorithmic considerations. The implementation of boundary conditions 
in deterministic algorithms can therefore be interpreted as using a prior, expressing 
an absolute belief in the selected boundary behavior. The added flexibility which 
characterizes statistical imaging methodologies makes it possible to import in the 
algorithm the postulated behavior of the image at the boundary with a certain degree 
of uncertainty. 

The distribution of gray levels within an image and the transition between 
areas with different gray-scale intensities are the most likely topics of a priori 
beliefs, hence primary targets for priors. In the nonstatistical imaging framework, 
a common choice of regularization, for the underlying least squares problems is 
a regularization functional, which penalizes growth in the norm of the derivative 
of the solution, thus discouraging solutions with highly oscillatory components. 
The corresponding statistical counterpart is a Markov model, based, for example, 
on the prior assumption that the gray-scale intensity at each pixel is a properly 
weighted average of the intensities of its neighbors plus a random innovation term 
which follows a certain statistical distribution. As an example, assuming a regular 
quadrilateral grid discretization, the typical local model can be expressed in terms of 
probability densities of pixel values X ; conditioned on the values of its neighboring 
pixels labeled according to their relative position to X; as Xup, Xdowns Xieft, and 
Xright, respectively. The conditional distribution is derived by writing 


X;|(Xup = Xup: X down = Xdown, Xett = Metts Xright = Xright) (5) 


1 
= Cu + Xdown + Xleft + Xright) + ; ; 


where ®,; is a random innovation process. For boundary pixels, an appropriate 
modification reflecting the a priori belief of the extension of the image outside the 
field of view must be incorporated. In a large variety of application, ®; is assumed 
to follow a normal distribution 


®; ~N (0,07), 


the variance 0; reflecting the expected deviation from the average intensity of the 
neighboring pixels. The Markov model can be expressed in matrix—vector form as 


LX =, 
where the matrix L is the five-point stencil discretization of the Laplacian in two 


dimensions and the vector ® € R% contains the innovation terms ® ;- AS we assume 
the innovation terms to be independent, the probability distribution of ® is 
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and the resulting prior model is a second-order Gaussian smoothness prior, 
1 
Tniw(s) cexp (—SE-"LxIP) 


Observe that the variances o; allow a spatially inhomogeneous a priori control of 
the texture of the image. Replacing the averaging weights 1/4 in (5) by more general 
weights p,;, 1 < k < 4 leads to a smoothness prior with directional sensitivity. 
Random draws from such anisotropic Gaussian priors are shown in Fig. 1, where 
each pixel with coordinate vector r; in a quadrilateral grid has eight neighboring 
pixels with coordinates us , and the corresponding weights p; are chosen as 


Fig. 1 Random draws from anisotropic Markov models. In the top row, the Markov model 
assumes stronger dependency between neighboring pixels in the radial than in angular direction, 
while in the bottom row, the roles of the directions are reversed. See text for a more detailed 
discussion 
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and the unit vector v; is chosen either as a vector pointing out of the center of the 
image (top row) or in a perpendicular direction (bottom row). The former choice 
thus assumes that pixels are more strongly affected by the adjacent values in the 
radial direction, while in the latter case, they have less influence than those in the 
angular direction. The factor t is added to make the matrix diagonally dominated. 

The just described construction of the smoothness prior is a particular instance 
of priors based on the assumption that the image is a Markov random field, (MRF). 
Similarly to the four-point average example, Markov random fields assume that the 
conditional probability distribution of a single pixel value X; conditioned on the 
remaining image depends only on the neighbors of X;, 


m (x; | xe, k Aj) =m (x; | xe E Nj), 


where N; is the list of neighbor pixels of X;, such as the four adjacent pixels in 
the model (5). In fact, the Hammersley—Clifford theorem (see [5]) states that prior 
distributions of MRF models are of the form 


N 
Tprior (x) OC exp | — > Vi(x) |. 


j=l 


where the function V;(x) depends only on x; and its neighbors. The simplest 
model in this family is a Gaussian white noise prior, where N; = @ and V;(x) = 
x7 /@o"), that is, 


1 
roe) exp (=I?) 


Observe that this prior assumes mutual independency of the pixels, which has 
qualitative repercussions on the images based on it. 

There is no theoretical reason to restrict the MRFs to Gaussian fields, and in fact, 
some of the non-Gaussian fields have had a remarkable popularity and success in the 
imaging context. Two non-Gaussian priors are particularly worth mentioning here, 
the ¢'-prior, where N; = @ and V; (x) = a@|x;|, that is, 


N 
prior (X) oc exp (—a||x|]1) , lx |l1 = x |x; |, 
j=l 
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and the closely related total variation (TV) prior, 


N 
Mprior(X) & exp(—ATV(x)), TV(x) = }) Vj (x), 


j=l 


with 


1 
Vj (x) = 3 > |x; — xx|. 


KEN; 


The former is suitable for imaging sparse images, where all but few pixels are 
believed to coincide with the background level that is set to zero. The latter prior 
is particularly suitable for blocky images, that is, for images consisting of piecewise 
smooth simple shapes. There is a strong connection to the recently popular concept 
of compressed sensing, see, for example, [11]. 

MRF priors, or priors with only local interaction between pixels, are by far the 
most commonly used priors in imaging. It is widely accepted and to some extent 
demonstrated (see [6] and the discussion in it) that the posterior density is sensitive 
to local properties of the prior only, while the global properties are predominantly 
determined by the likelihood. Thus, as far as the role of priors is concerned, it 
is important to remember that until the likelihood is taken into account, there is 
no connection with the measured data, hence no reason to believe that the prior 
should generate images that in the large scale resemble what we are looking for. 
In general, priors are usually designed to carry very general often qualitative and 
local information, which will be put into proper context with the guidance of the 
data through the integration with the likelihood. To demonstrate the local structure 
implied by different priors, in Fig. 2, we show some random draws from the priors 
discussed above. 


Likelihood: Forward Model and Statistical Properties of Noise 


If an image is worth a thousand words, a proper model of the noise corrupting it 
is worth at least a thousand more, in particular when the processing is based on 
the statistical methods. So far, the notion of noise has remained vague, and its 
role unclear. It is the noise, and in fact its statistical properties, that determines the 
likelihood density. We start by considering two very popular noise models. 

Additive, nondiscrete noise: An additive noise model assumes that the data and 
the unknown are in a functional relation of the form 


y = F(x) +e, (6) 


where e is the noise vector. If the function F is linear, or it has been linearized, the 
problem simplifies to 
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Fig. 2. Random draws from various MRF priors. Top row: white noise prior. Middle row: sparsity 
prior or £!-prior with positivity constraint. Bottom row: total variation prior 


y=Axte. (7) 


The stochastic extension of (6) is 
Y=F(X)+E, 


where Y, X, and E are multivariate random vectors. 

The form of the likelihood is determined not only by the assumed probability 
distributions of Y, X, and E but also by the dependency between pairs of these 
variables. In the simplest case, X and E are assumed to be mutually independent 
and the probability density of the noise vector known, 


E ~ Thoise (e) , 
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resulting in a likelihood function of the form 


m(y | x) & Mnoise (y _ F(x)) ’ 


which is one of the most commonly used in applications. A particularly popular 
model for additive noise is a Gaussian noise, 


E~N(0,%), 
where the covariance matrix & is positive definite. Therefore, if we write yo = 


D'D, where D can be the Cholesky factor of 5~! or D = 5~!/?, the likelihood can 
be written as 


m(y | x) & exp (-30 = F@)TZ"O- F(x) 
= exp(- FIDO - Fel), (8) 


In the general case where X and E are not independent, we need to specify the joint 
density 


(X, E) ~ m(x,e) 
and the corresponding conditional density 


u(x, e) 


Iv prior (x ) j 


noise (€ | x) = 


In this case, the likelihood becomes 
m(y | X) & Mnoise(y — F(x) | x). 


This clearly demonstrates the problems which may arise if we want to adhere 
to the claim that “likelihood should be independent of the prior.’ Because the 
interdependency of the image x and the noise is much more common than we might 
be inclined to believe, the independency of noise and signal is often in conflict with 
reality. An instance of such situation occurs in electromagnetic brain imaging using 
magnetoencephalography (MEG) or electroencephalography (EEG), when the eye 
muscle during a visual task acts as noise source but can hardly be considered as 
independent from the brain activation due to a visual stimulus. Another example 
related to boundary conditions will be discussed later on. Also, since the noise 
term should account not only for the exogenous measurement noise but also for 
the shortcomings of the model, including discretization errors, the interdependency 
is in fact a ubiquitous phenomenon too often neglected. 
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Most additive noise models assume that the noise follows a Gaussian distribution, 
with zero mean and given covariance. The computational advantages of a Gaussian 
likelihood are rather formidable and have been a great incentive to use Gaussian 
approximations of non-Gaussian densities. While it is commonplace and somewhat 
justified, for example, to approximate Poisson densities with Gaussian densities 
when the mean is sufficiently large [14], there are some important imaging applica- 
tions where the statistical distribution of the noise must be faithfully represented in 
the likelihood. 

Counting noise: The weakness of a signal can complicate the deblurring and 
denoising problem, as is the case in some image processing applications in 
astronomy [49, 57, 63], microscopy [45, 68], and medical imaging [29, 60]. In fact, 
in the case of weak signals, a charge-coupled device (CCD), instead of recording 
an integrated signal over a time window, counts individual photons or electrons. 
This leads to a situation where the noise corrupting the recorded signal is no longer 
exogenous but rather an intrinsic property of the signal itself, that is, the input 
signal itself is a random process with an unpredictable behavior. Under rather mild 
assumptions — stationarity, independency of increments, and zero probability of 
coincidence — it can be shown (see, e.g., [62]) that the counting signal follows a 
Poisson distribution. Consider, for example, the astronomical image of a very distant 
object, collected with an optical measurement device whose blurring is described 
by a matrix A. The classical description of such data would follow (7), with the 
error term collecting the background noise and the thermal noise of the device. The 
corresponding counting model is 


y; ~ Poisson ((Ax); +b), yj, yx independent if j # k, 


or, explicitly, 


“ ((Ax); +b) 
m(y |x) = [] SO xp (Any; +5). 
oI 


j=l 


where b > 0 is a background radiation level, assumed known. Observe that while 
the data are counts, therefore integer numbers, the expectation need not to be. 

Similar or slightly modified likelihoods can be used to model the positron 
emission tomography (PET) and single-photon emission computed tomography 
(SPECT) signals; see [29,54]. 

The latter example above demonstrates clearly that the description of imaging 
problems as linear or nonlinear, without a specification of the noise model, in 
the context of statistical methods, does not play a significant role: Even if the 
expectation is linear, traditional algorithms for solving linear inverse problems are 
useless, although they may turn out to be useful within iterative solvers for solving 
locally linearized steps. 
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Maximum Likelihood and Fisher Information 


When switching to a parametric non-Bayesian framework, the statistical inference 
problem amounts to estimating a deterministic parameter that identifies the proba- 
bility distribution from which the observations are drawn. To apply this framework 
in imaging problems, the underlying image x, which in the Bayesian context was 
itself a random variable, can be thought of as a parameter vector that specifies the 
likelihood function, 


S(y; x) = r(y | x), 


as implied by the notation f(y; x) also. 

In the non-Bayesian interpretation, a measure of how much information about 
the parameter x is contained in the observation is given in terms of the Fisher 
information matrix J, 


SQy;x)dy. (9) 


ee 


Ox; OXk Ox; OXk 


In this context, the observation y only is a realization of a random variable Y , whose 
probability distribution is entirely determined by the distribution of the noise. The 
gradient of the logarithm of the likelihood function is referred to as the score, and 
the Fisher information matrix is therefore the covariance of the score. 

Assuming that the likelihood is twice continuously differentiable and regular 
enough to allow the exchange of integration and differentiation, it is possible to 
derive another useful expression for the information matrix. It follows from the 
identity 


cml i) 
oe (10) 
OX; f OX; 
that we may write the Fisher information matrix as 
dlog f of 0 dlog f 0’ log f 
Jin = dy = dy — d 
ih Ox; OXK OX Ox; fay OX j OXK fay 
Using the identity (10) with & replaced by 7, we observe that 
dlo 
jen fay = f shay = 5 | fay =o 
since the integral of f is one, which leads us to the alternative formula 
07 log f 07 log f 
Jip=r =— ; 11 
ue ay dee 4 mie oa 
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The Fisher information matrix is closely related to non-Bayesian estimation theory. 
This will be discussed later in connection with maximum likelihood estimation. 


Informative or Noninformative Priors? 


Not seldom the use of priors in imaging applications is blamed for biasing the 
solution in a direction not supported by the data. The concern of the use of committal 
priors has led to the search of “noninformative priors” [39] or weak priors that would 
“let the data speak.” 

The strength or weakness of a prior is a rather elusive concept, as the importance 
of the prior in Bayesian imaging is in fact determined by the likelihood: the more 
information we have about the image in data, the less has to be supplied by the 
prior. On the other hand, in imaging applications where the likelihood is built on 
very few data points, the prior needs to supply the missing information, hence has 
a much more important role. As pointed out before, it is a common understanding 
that in imaging applications, prior should carry small-scale information about the 
image that is missing from the likelihood that in turn carries information about the 
large-scale features and in that sense complements the data. 


Adding Layers: Hierarchical Models 
Consider the following simple denoising problem with additive Gaussian noise, 
Y=X+N, N~N(,%), 


with noise covariance matrix & presumed known, whose likelihood model is 
tantamount to saying that 


Y|X =x~WN(x,%). 


From this perspective, the denoising problem is reduced to estimating the mean 
of a Gaussian density in the non-Bayesian spirit, and the prior distribution is a 
hierarchical model, expressing the degree of uncertainty of the mean x. 

Parametric models are common when defining the prior densities, but similarly 
to the above interpretation of the likelihood, the parameters are often poorly known. 
For example, when introducing a prior 


X ~N(6,P) 


with unknown 0, we are expressing a qualitative prior belief that “X differs from 
an unknown value by an error with a given Gaussian statistics,’ which says very 
little about the values of X itself unless information about 6 is provided. Similarly 
as in the denoising problem, it is natural to augment the prior with another layer of 
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information concerning the parameter 0. This layering of the inherent uncertainty 
is at the core of hypermodels, or Bayesian hierarchical models. Hierarchical models 
are not restricted to uncertainties in the prior, but can be applied to lack of 
information of the likelihood model as well. 

In hierarchical models, both the likelihood and the prior may depend on 
additional parameters, 


m(y|x) >a | x,y), Tprior (X) =e Tprior (X | 0), 


with both parameters y and @ poorly known. In this case, it is natural to augment 
the model with hyperpriors. Assuming for simplicity that the parameters y and 0 
are mutually independent so that we can define the hyperprior distributions 7 (y) 
and 72(@), the joint probability distribution of all the unknowns is 


a(x, y, 9,7) = m(y | X,Y) Hprior (x | 0)71(y)m2(0). 


From this point on, the Bayesian inference can proceed along different paths. It is 
possible to treat the hyperparameters as nuisance parameters and marginalize them 
out by computing 


(x,y) =f [my y)d0dy 


and then proceed as in a standard Bayesian inference problem. Alternatively, the 
hyperparameters can be included in the list of unknowns of the problem and their 
posterior density 


y. 8, . 
ng |y) = Pe, c= | 9 
Y 


needs to be explored. The estimation of the hyperparameters can be based on the 
optimization or on the evidence, as will be illustrated below with a specific example. 
To clarify the concept of a hierarchical model itself, we consider some examples 
where hierarchical models arise naturally. 
Blind deconvolution: Consider the standard deblurring problem defined in sec- 
tion “Imaging Problems.” Usually, it is assumed that the blurring kernel A is known, 
and the likelihood, with additive Gaussian noise with covariance &, becomes 


m(y | x) « exp (-50 Antsy An) : (12) 


In some cases, although A is poorly known, its parametric expression is known and 
the uncertainty only affects the values of some parameters, as is the case when the 
shape of the continuous convolution kernel a(r — 5) is known but the actual width 
is not. If we express the kernel a as a function of a width parameter, 
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1 
a(r—s) =a,(r—s) = pny eowh y>0, 


and denote by A, the corresponding discretized convolution matrix, the likelihood 
becomes 


1 
m(y | x,y) « exp (-30 —A,x)TI"(y - A,x)) : 


and additional information concerning y, for example, bound constraints, can be 
included via a hyperprior density. 

The procedure just outlined can be applied to many problems arising from 
adaptive optics imaging in astronomy [52]; while the uncertainty in the model is 
more complex than in the explanatory example above, the approach remains the 
same. 

Conditionally Gaussian hypermodels: Gaussian prior models are often criticized 
for being a too restricted class, not being able to adequately represent prior beliefs 
concerning, for example, the sparsity or piecewise smoothness of the solution. The 
range of qualitative features that can be expressed with normal densities can be 
considerably expanded by considering conditionally Gaussian families instead. As 
an example, consider the problem of finding a sparse image from linearly blurred 
noisy copy of it. The likelihood model in this case may be written as in (12). To set 
up an appropriate prior, consider a conditionally Gaussian prior 


1 1/2 N 
j 
rnin | 8) & (5 =) exp ae 3, 
j=l 
NT y2 
1 ss 
= a —+log6;|]. 1 
exp |e +e | (13) 


If 0; = 6 = constant, we obtain the standard white noise prior which cannot be 
expected to favor sparse solutions. On the other hand, since 6; is the variance of the 
pixel X;, sparse images correspond to vectors 6 with most of the components close 
to zero. Since we do not know a priori which of the variances should significantly 
differ from zero, when choosing a stochastic model for 6, it is reasonable to select 
a hyperprior that favors sparsity without actually specifying the location of the 
outliers. Two distributions that are particularly well suited for this are the gamma 
distribution, 


6, 
6; ~ Gamma(k, 6), k,0>0, 2(6;)= 6! lexp (-2) 
i) 


and the inverse gamma distribution, 
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6 
0; ~ InvGamma(k, 4%), k,%>0, 2(6;)= 6-*lexp (-3) 
J 


The parameters k and 6p are referred to as the shape and the scaling, respectively. 
The inverse gamma distribution corresponds to assuming that the precision, defined 
as 1/6;, is distributed according to the gamma distribution Gamma(k, 1/6). The 
computational price of introducing hyperparameters is that instead of one image 
x, we need to estimate the image x and its variance image @. Fortunately, for 
conditionally Gaussian families, there are efficient algorithms for computing these 
estimates, which will be discussed in the section concerning algorithms. 

The hyperprior based on the gamma distribution, in turn, contains parameters 
(k and 6) to be determined. Nothing prevents us from defining another layer of 
hyperpriors concerning these values. It should be noted that in hierarchical models, 
the selection of the parameters higher up in the hierarchy tends to have less direct 
effect on the parameters of primary interest. Since this last statement has not been 
formally proved to be true, it should be considered as a piece of computational 
folklore. 

Conditionally Gaussian hypermodels have been successfully applied in machine 
learning [66], in electromagnetic brain activity mapping [16], and in imaging 
applications for restoring blocky images [15]. Recently, their use in compressed 
sensing has been proposed [40]. 


4 Numerical Methods and Case Examples 


The solution of an imaging inverse problem in the statistical framework is the 
posterior probability density. Because this format of the solution is not practical 
for most applications, it is common to summarize the distribution in one or a 
few images. This leads to the challenging problem of exploring the posterior 
distributions and finding single estimators supported by the distribution. 


Estimators 


In this section, we review some of the commonly used estimators and subsequently 
discuss some of the popular algorithms suggested in the literature to compute the 
corresponding estimates. 


Prelude: Least Squares and Tikhonov Regularization 

In the case where the forward model is linear, the problem of estimating an image 
from a degraded, noisy recording is equivalent in a determinist setting to looking for 
a solution of a linear system of equations of the form 


Ax = y, (14) 
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where the right-hand side is corrupt by noise. When A is not a square matrix 
and/or it is ill conditioned, one needs to specify what a “solution” means. The most 
straightforward way is to specify it as a least squares solution. 

There is a large body of literature, and a wealth of numerical algorithms, for 
the solution of large-scale least squares problems arising from problems similar 
to imaging applications (see, e.g., [9]). Since dimensionality alone makes these 
problems computationally very demanding, they may require an unreasonable 
amount of computer memory and operations unless a compact representation of 
the matrix A can be exploited. Many of the available algorithms make additional 
assumptions about either the underlying image or the structure of the forward model 
regardless of whether there is a good justification. 

In a determinist setting, the entries of the least squares solution of (14) with a 
right-hand side corrupted by noise are not necessarily in the gray-scale range of 
the image pixels. Moreover, the inherent ill conditioning of the problem, which 
varies with the imaging modality and the conditions under which the observations 
were collected, usually requires reqularization, see, for example, [4, 33, 34, 41]. 
A standard regularization method is to replace the original ill-posed least squares 
problem by a nearby well-posed problem by introducing a penalty term to avoid 
that the computed solution is dominated by amplified noise components, reducing 
the problem to minimizing a functional of the form 


T(x) = ||Ax — yl? + aJ(x), (15) 


where J(x) is the penalty functional and a > 0 is the regularization parameter. The 
minimizer of the functional (15) is the Tikhonov regularized solution. The type of 
additional information used in the design of the penalty term may include upper 
bounds on the norm of the solution or of its derivatives, nonnegative constraints for 
its entries, or bounds on some of the components. Often, expressing characteristics 
that are expected of the sought image in qualitative terms is neither new nor difficult: 
the translation of these beliefs into mathematical terms and their implementation is 
a more challenging step. 


Maximum Likelihood and Maximum A Posteriori 

We begin with the discussion of the maximum likelihood estimator in the framework 
of non-Bayesian statistics and denote by x a deterministic parameter determining 
the likelihood distribution of the data, modeled as a random variable. Let X = X(y) 
denote an estimator of x, based on the observations y. Obviously, X is also a random 
variable, because of its dependency on the stochastic observations y; moreover, it is 
an unbiased estimator if 


Eix(y)} = x, 


that is, if, in the average, it returns the exact value. The covariance matrix C of an 
unbiased estimator therefore measures the statistical variation around the true value, 
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Cie = E{(Rj — x) Gx —xe)}, 


thus the name mean square error. Evidently, the smaller the mean square error, for 
example, in the sense of quadratic forms, the higher the expected fidelity of the 
estimator. The Fisher information matrix (9) gives a lower bound for the covariance 
matrix of all unbiased estimators. Assuming that J is invertible, the Cramér—Rao 
lower bound states that for an unbiased estimator, 


Jt<c 
in the sense of quadratic forms, that is, for any vector 
Ol ew Cx 


An estimator is called efficient if the error covariance reaches the Cramér—Rao 
bound. 

The maximum likelihood estimator Xy41(y) is the maximizer of the function x t— 
f(x; y), and in practice, it is found by locating the zero(s) of the score, 


Vx log f(x; y) =0 > x = km_(). 


Notice that in the non-Bayesian context, likelihood refers solely to the likelihood 
of the observations y, and the maximum likelihood estimation is a way to choose the 
underlying parametric model so that the observations become as likely as possible. 

The popularity of the maximum likelihood estimator, in addition to being an 
intuitively obvious choice, stems from the fact that it is asymptotically efficient 
estimator in the sense that when the number of independent observations of the data 
increases, the covariance of the estimator converges toward the inverse of the Fisher 
information matrix, assuming that it exists. More precisely, assuming a sequence 
y!, y?,... of independent observations and defining £” = % (y', oes y") as 


io 
An __ = J 
<" = argmax 4 ~ Lf (x,y’)P, 


asymptotically the probability distribution of x” approaches a Gaussian distribution 
with mean x and covariance J~!. 

The assumption of the regularity of the Fisher information matrix limits the use 
of the ML estimator in imaging applications. To understand this claim, consider the 
simple case of linear forward model and additive Gaussian noise, 


Y=Ax+E, E~N(0,3). 


The likelihood function in this case is 
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1 1/N 1 ‘ ; 
f(xy) = (sm) exp (-30 —Ax) xy -Ax)) : 


from which it is obvious that by formula (11), 
J=ATEA. 


In the simplest imaging problems such as of denoising, the invertibility of J is not an 
issue. However, in more realistic and challenging applications such as deblurring, 
the ill conditioning of A renders J singular, and the Cramér—Rao bound becomes 
meaningless. It is not uncommon to regularize the information matrix by adding a 
diagonal weight to it which, from the Bayesian viewpoint, is tantamount to adding 
prior information but in a rather uncontrolled manner. 

For further reading of mathematical methods in estimation theory, we refer to 
[17,46, 50]. 

We consider the maximum likelihood estimator in the context of regularization 
and Bayesian statistics. In the case of a Gaussian additive noise observation model, 
under the assumption that the noise at each pixel is independent of the signal and 
that the forward map is linear, F(x) = Ax, the likelihood (8) is of the form 


1 
m(y |x) ocexp(—S|ID(Ax— ») IP). 


where » is the noise covariance matrix and D'D = 7! is the Cholesky 
decomposition of its inverse. The maximizer of the likelihood function is the 
solution of the minimization problem 


xm_ = argmin {| D(Ax - y)I?} , 
which, in turn, is the least squares solution of the linear system 
DAx = Dy. 


Thus, we can reinterpret least squares solutions as maximum likelihood esti- 
mates under an additive, independent Gaussian error model. Within the statistical 
framework, the maximum likelihood estimator is defined analogously for any error 
model which admits a maximizer for the likelihood, but in the general case, the 
computation of the minimizer cannot be reduced to the solution of a linear least 
squares problem. 

In a statistical framework, the addition of a penalty terms to keep the solution of 
the least squares problem from becoming dominated by amplified noise components 
is tantamount to using a prior to augment the likelihood. If the observation model is 
linear, the prior and the likelihood are both Gaussian, 
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1 
Tprior(X) O& Exp (-"r"'x} ; 
and the noise is independent of the signal, the corresponding posterior is of the form 
1 
n(x | y) ocexp (=> (ID(Ax — y)I? + IRIP)) 


where R satisfies R'R = I~, so typically it is the Cholesky factor of [~! or 
alternatively, R = [~!/?, 

The maximizer of the posterior density, or the maximum a posteriori (MAP) 
estimate, is the minimizer of the negative exponent, hence the solution of the 
minimization problem 


xmap = argmin{||D(Ax — y)||? + ||Rx||7} 
= argmin ois x Dy 
ae R 0 


or, equivalently, the Tikhonov solution (15) with penalty J(x) = ||Rx||* and 
regularization parameter a = 1. Again, it is important to note that the direct 
correspondence between the Tikhonov regularization and the MAP estimate only 
holds for linear observation models and Gaussian likelihood and prior. The fact that 
the MAP estimate in this case is the least squares solution of the linear system 


DA Dy 
= 16 
als=[o) as 
is a big incentive to stay with Gaussian likelihood and Gaussian priors as long as 
possible. 
As in the case of the ML estimate, the definition of MAP estimate is independent 
of the form of the posterior, hence applied also to non-Gaussian, nonindependent 


noise models, with the caveat that in the general case, the search for a maximizer of 
the posterior may require much more sophisticated optimization tools. 


Conditional Means 

The recasting in statistical terms of imaging problems effectively shifts the interest 
from the image itself to its probability density. The ML and MAP estimators 
discussed in the previous section suffer from the limitations, which come from 
summarizing an entire distribution with one realization. The ML estimator is known 
to suffer from instabilities due to the typical ill conditioning of the forward map in 
imaging problems, and it will not be discussed further here. The computed MAP 
estimate, on the other hand, may correspond to an isolated spike in the probability 
density away from the bulk of the mass of the density, and its computation may 
suffer from numerical complications. Furthermore, a conceptually more serious 
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limitation is the fact that MAP estimators do not carry information about the 
statistical dispersion of the distribution. A tight posterior density suggests that 
any ensemble of images which are in statistical agreement with the data and the 
given prior show little variability; hence, any realization from that ensemble can 
be thought of as very representative of the entire family. A wide posterior, on 
the other hand, suggests that there is a rather varied family of images that are in 
agreement with the data and the prior, hence lowering the representative power of 
any individual realization. 

In the case where either the likelihood or the prior is not Gaussian, the mean 
of the posterior density, often referred to as conditional mean (CM) or posterior 
mean, may be a better choice because it is the estimator with least variance (see [3, 
41]). Observe, however, that in the fully Gaussian case, the MAP and CM estimate 
coincides. 

The CM estimate is, by definition, 


rom fsnte | nae 
RN 


while the a posteriori covariance matrix is 


Tom = ki (x — xem) (x — xem) "a(x | y)dx, 


hence requiring the evaluation of the high-dimensional integrals. When the integrals 
have no closed form solution, as is the case for many imaging problems where, 
for example, the a priori information contains bounds on pixel values, a numerical 
approximation of the integral must be used to estimate xcy and cm. The large 
dimensionality of the parameter space, which easily is of the order of hundreds 
of thousands when x represents an image, rules out the use of standard numerical 
quadratures, leaving Monte Carlo integration the only currently known feasible 
alternative. 

The conceptual simplicity of Monte Carlo integration, which estimates the 
integral value as the average of a large sample of the integrand evaluated over the 
support of the integration, requires a way of generating a large sample from the 
posterior density. The generation of a sample from a given distribution is a well- 
known problem in statistical inference, which has inspired families of sampling 
schemes generically referred to as Markov chain Monte Carlo (MCMC) methods, 
which will be discussed in section “Markov Chain Monte Carlo Sampling.” 

Once a representative sample from the posterior has been generated, the CM 
estimate is approximately the sample mean. By definition, the CM estimate must be 
near the bulk of the density, although it is not necessarily a highly probable point. 
In fact, for multimodal distributions, the CM estimate may fall between the modes 
of the density and even belong to a subset of R% with probability zero, although 
such a situation is rather easy to detect. There is evidence, however, that in some 
imaging applications the CM estimate is more stable than the MAP estimate; see 
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[23]. While the robustness of the CM estimate does not compensate for the lack 
of information about the width of the posterior, the possibility of estimating the 
posterior covariance matrix via sampling is an argument for the sampling approach, 
since the sample can also be used to estimate the posterior width. 


Algorithms 


The various estimators based on the posterior distribution are simple to define, but 
the actual computation may be a major challenge. In the case of Gaussian likelihood 
and prior, combined with linear forward map, the MAP and CM estimates coincide 
and an explicit formula exists. If the problem is very high dimensional, even this 
case may be computationally challenging. Before going to specific algorithms, we 
review the linear Gaussian theory. 

The starting point is the linear additive model 


Y=AX+E, X~N(0,T), E~N(0,%). 
Here, we assume that the mean of X and the noise F both vanish, an assumption 


that is easy to remove. Above, X and F need not be mutually independent, and we 
may postulate that they are jointly Gaussian and the cross-correlation matrix 


C=E{XE"} e RY*™ 


may not vanish. The joint probability distribution of X and Y is also Gaussian, with 
zero mean and variance 


4 _ XX X(AX + E)" 
. 7 [x" ‘all = ae + E)X™ (AX + E)(AX + Pat 


= ro TAT+C 
~ 1AT+C! ATAT+ DI" 


Let L ¢ RY+)*(+™) denote the inverse of the above matrix, assuming that it 
exists, and write a partitioning of it in blocks according to the dimensions N and M, 


as r raAta4c a [balks 
AY +C'ATAT+ > Loy Loo} 


With this notation, the joint probability distribution of X and Y is 


1 
w(x, y) « exp (-po7lnx + x"Lyy + y Lox + y"Lay)) ‘ 
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To find the posterior density, one completes the square in the exponent with respect 
to x, 


1 
w(x | y) exp (-; (x —Ly'Lny)' Lu (x - LiiLny)) : 


where terms independent of x that contribute only to the normalization are left out. 
Therefore, 


X|¥=y~N (U'Lvy, 7). 


Finally, we need to express the matrix blocks L;; in terms of the matrices of 
the model. The expressions follow from the classical matrix theory of Schur 
complements [24]: We have 


f= = (PA! 4) (ATA +5) (AT 4’), (17) 
and 
LilLioy = (TAT +.C) (APAT + 5) y. (18) 


Although a closed form solution, to evaluate the expression (18) for the posterior 
mean may require iterative solvers. 

When the image and the noise are mutually independent, implying that C = 0, 
we find a frequently encountered form of the MAP estimate arising from writing the 
Gaussian posterior density directly by using Bayes’ formula, that is, 


(x | y) ax prior (x) 0 (y | x) 
1 1 
x exp (-ps1r's - 5 =fay > = Ax)) ; 
and so the MAP estimate, and simultaneously the posterior mean estimate, is the 


maximizer of the above expression, or, equivalently, the minimizer of the quadratic 
functional 


H(x) = (y —Ax)TT'(y — Ax) + xT. 
By substituting the factorizations 
= !=p'D, rr! =R'R, 


the minimization problem becomes the previously discussed standard least squares 
problem of minimizing 


H(x) = ||D(v — Ax) ||? + ||Rx|/’, (19) 
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leading to the least squares problem (16). Whether one should use this formula or 
(18) depends on the application and, in particular, on the sparsity properties of the 
covariance matrices and their inverses. 


Iterative Linear Least Squares Solvers 
The computation of the ML or MAP estimate under the Gaussian additive linear 
noise model and, in the latter case, with a Gaussian prior, amounts to the solution 
of system of linear equations (14), (16), or (18) in the least squares sense. Since the 
dimensions of the problem are proportional to the number of pixels in the image 
except when the observation model has a particular structure or sparsity properties 
which can be exploited to reduce the memory allocation, solution by direct methods 
is unfeasible, hence making in general the iterative solvers the methods of choice. 

Among the iterative methods specifically designed for the solution of least 
squares problems, the LSQR version with shifts [55,56] of the Conjugate Gradient 
for Least Squares (CGLS) method originally proposed in [37] combines robustness 
and numerical efficiency. CGLS-type iterative methods have been designed to solve 
the system Ax = y, minimize ||Ax — y||*, or minimize ||Ax — y||? + 6]|x|l?, 
where the matrix A may be square or rectangular — either overdetermined or 
underdetermined — and may have any rank. The matrix A does not need to be 
stored, but instead its action is represented by a routine for computing matrix—vector 
products of the forms v +> Av and ut> Alu. 

Minimizing the expression (19) may be transformed in a standard form by 
writing it as 


min {||D (y — AR“) |? + |lwI?}, w= Rx 


In practice, the matrix R7! should not be computed, unless it is trivial to obtain. 
Rather, R~! acts as a preconditioner, and its action should be implemented together 
with the action of the matrix A as a routine called from the iterative linear solver. 
The interpretation of the action of the prior as a preconditioner has led to the concept 
of prior conditioner; see [12, 14] for details. 


Nonlinear Maximization 

In the more general case where either the observation model is nonlinear or the 
likelihood and prior are non-Gaussian, the computation of the ML and MAP 
estimates requires the solution of a maximization problem. Maximizers of nonlinear 
functions can be found by quasi-Newton methods with global convergence strategy. 
Since Newton-type methods proceed by solving a sequence of linearized problems 
whose dimensions are proportional to the size of the image, iterative linear 
solvers are typically used for the solution of the linear subproblem [20, 43]. In 
imaging applications, it is not uncommon that the a priori information includes 
nonnegativity constraints on the pixel values or bounds on their range. In these 
cases, the computation of the MAP estimate amounts to a constrained maximization 
problem and may be very challenging. Algorithms for maximization problems with 
nonnegativity constraints arising in imaging applications based on the projected 
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gradient have been proposed in the literature; see [2] and references therein. We 
shall not review Newton-based methods here, since usually the fine points are related 
to the particular applications at hand and not so much to the statistical description 
of the problem. Instead, we review some algorithms that stem directly from the 
statistical setting of the problem and are therefore different from the methods used 
in regularized deterministic literature. 


EM Algorithm 
The MAP estimator is the maximizer of the posterior density z(x | y), or, 
equivalently, the maximizer the logarithm of it, 


L(x | y) = log x(x | y) = log x(y | x) + log zprior(x) + constant, 


where the simplest form of Bayes’ rule was used to represent the posterior density 
as a product of the likelihood and the prior. However, note that above, the vector x 
may represent the unknown of primary interest, or if hierarchical models are used, 
the model parameters related to the likelihood and/or prior may be included in it. 

The expectation—maximization algorithm is a method developed originally for 
maximizing the likelihood function and later extended to the Bayesian setting to 
maximize the posterior density, in a situation where part of the data is “missing.” 
While in many statistical application the concept of missing data appears natural, 
for example, when incomplete census data or patient data are discussed, in imaging 
applications, this concept is a rather arbitrary and to some extent artificial. However, 
during the years, EM has found its way to numerous imaging applications, partly 
because it often leads to algorithms that are easy to implement. Early versions of the 
imaging algorithms with counting data such as the Richardson—Lucy iteration [49, 
57], popular in astronomical imaging, were independently derived. Later, similar 
EM-based algorithms were rederived in the context of medical imaging [29, 36, 60]. 
Although EM algorithms are discussed in more detail elsewhere in this book, we 
include a brief discussion here in order to put EM in the context of general statistical 
imaging formalism. 

As pointed out above, in imaging problems, data is not missing: Data, per 
definitionem, is what one is able to observe and register. Therefore, the starting 
point of the EM algorithm in image applications is to augment the actual data y 
by fictitious, nonexistent data z that would make the problem significantly easier to 
handle. 

Consider the statistical inference problem of estimating a random variable X 
based on an observed realization of Y, denoted by Y = y = yops. We assume 
the existence of a third random variable Z and postulate that the joint probability 
density of these three variables is available and is denoted by (x, y, z). The EM 
algorithm consists of the following steps: 


1. Initialize x = x° and set k = 0. 
2. E-step: Define the probability distribution, or a fictitious likelihood density, 
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m*(z) =x (z|x*,y) xa (x*,y,z), y = Yobs, 


and calculate the integral 
Ox) =f Levly.dat@dz, L6e|y.2) =log(ee|y.9)- 20) 


3. M-step: Update x* by defining 
k+l _ k 
x = argmax QO" (x). (21) 


4. If a given convergence criterion is satisfied, exit; otherwise, increase k by one 
and repeat from step 2 until convergence. 


The E-step above can be interpreted as computing the expectation of the real- 
valued random variable log(z(x, y, Z)), x and y fixed, with respect to a conditional 
measure of Z conditioned on X = x/ and Y = Y = Yobs, hence the name 
expectation step. 

The use of the EM algorithm is often advocated on the basis of the convergence 
proof given in [19]. Unfortunately, the result is often erroneously quoted as an 
automatic guarantee of convergence, without verifying the required hypotheses. 
The validity of the convergence is further obfuscated by the error in the proof 
(see [70]), and in fact, counterexamples of lack of convergence are well known 
[10,69]. We point out that as far as convergence is concerned, global convergence 
of quasi-Newton algorithm is well established, and compared to the EM algorithm, 
the algorithm is often more effective [20]. 

As the concept of missing data is not well defined in general, we outline the use 
of the EM algorithm in an example that is meaningful in imaging applications. 

SPECT imaging: The example discussed here follows the article [29]. Consider 
the SPECT image formation problem, where the two-dimensional object is divided 
in N pixels, each one emitting photons that are recorded through collimators by 
M photon counting devices. If x; is the expected number of photons emitted by the 
jth pixel, the photon count at ith photon counter, denoted by Y;, is an integer-valued 
random variable and can be modeled by a Poisson process, 


M 
Y; ~ Poisson Se aij x; = Poisson((Ax);), 
j=l 


the variables Y; being mutually independent and the matrix elements a;j of A € 


R™”*"% being known. We assume that X, the stochastic extension of the unknown 
vector x € RY, is a priori distributed according to a certain probability distribution, 


X~ Tprior (X) x exp(—V(x)). 
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To apply the EM algorithm, we need to decide how to define the “missing data.” 
Photon counter devices detect the emitted photons added over the line of sight; 
evidently, the problem would be more tractable if we knew the number of emitted 
photons from each pixel separately. Therefore, we define a fictitious measurement, 


Zi; ~ Poisson(a;;x;), 


and posit that these variables are mutually independent. Obviously, after the 
measurement Y = y, we have 


N 
ez Zij = Yi- (22) 


To perform the E-step, assuming that x‘ is given, consider first the conditional 
density x*(z) = m (z| x*, y). 

A basic result from probability theory states that if NM independent random 
variables A; are a priori Poisson distributed with respective means ;, and in 
addition 


then, a posteriori, the variables A; conditioned on the above data are binomially 
distributed, 


N 
Aj |[SlA;=K ~ Bin (x, 4). 
j=l ja hy 


In particular, the conditional expectation of A ; is 


We therefore conclude that the conditional density 2‘ (z) is a product of binomial 
distributions of Z;; with a priori means 4; = aijx', ae Lj = (Ax*);, and 
K = y;, so in particular, 


N 

Gig tk 42 

E Zij | ) Lij = Ji oe (z)dz= yi nh, 22d. (23) 
j=l 
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Furthermore, by Bayes’ theorem, 
m(X | y,z) = W(x | z) = W(zZ | X)Aprior(X), 


where we used the fact that the true observations y add no information on x that 
would not be included in z, we have, by definition of the Poisson likelihood and the 
prior, 


L(x | y,Z) = >. (zi; log(ajjx;) = dij X;) = V(x) + constant, 
ij 


and therefore, up to an additive constant, we have 


OF (x) = > (zi log (aij xj) —aix;) — V(x), 


1] 


where Z is defined in (23). This completes the E-step. 


The M-step requires the minimization of Q*(x) given above. Assuming that V 
is differentiable, the minimizer should satisfy 


m 


1 aV 
at = Y= aie = ax = 0. 


i=1 i=1 


How complicated it is to find a solution to this condition depends on the prior con- 
tribution V and may require an internal Newton iteration. In [29], an approximate 
“one-step late” (OSL) algorithm was suggested, which is tantamount to a fixed-point 
iteration: Initiating with X° = x‘, an update scheme x! — X't! is given by 


m k 

gitl — ini Ze 
t ~~ aV (=ty’ 
Via aie + i (XD 


and this step is repeated until a convergence criterion is satisfied at some t = fx. 
Finally, the M-step is completed by updating xt! = x'*. 
The EM algorithm has been applied to other imaging problems such as blind 


deconvolution problem [44] and PET imaging [36, 71]. 


Markov Chain Monte Carlo Sampling 

In Bayesian statistical imaging, the real solution of the imaging problem is the 
posterior density of the image interpreted as a multivariate random variable. If a 
closed form of the posterior is either unavailable or not suitable for the tasks at hand, 
the alternative is to resort to exploring the density by generating a representative 
sample from it. Markov chain Monte Carlo (MCMC) samplers yield samples from 
a target distribution by moving from a point in a chain to the next by the transition 
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rule which characterizes the specific algorithm. MCMC sampling algorithms are 
usually subdivided into those which are variants of the Metropolis—Hastings (MH) 
algorithm or the Gibbs sampler. While the foundations of the MH algorithm were 
laid first [25, 35,51], Gibbs samplers have sometimes the appeal of being more 
straightforward to implement. 

The basic idea of Monte Carlo integration is rather simple. Assume that (x) is 
a probability density in R’, and let {x!, Pe cae nf denote a stochastic process, 
where the random variables X’ are independent and identically distributed, X’ ~ 
a(x). The central limit theorem asserts that for any measurable f : RY > R, 


> F(x") =e f Ft (x)a(x)dx almost certainly, (24) 
i=l RN 

and moreover, the convergence takes place asymptotically with the rate 1/./n, 
independently of the dimension N. The difficulty is to find a computationally 
efficient way of drawing independently from a given distribution 2. Indeed, when 
N is large, it may be even difficult to decide where the numerical support of the 
density is. In MCMC methods, instead of producing an independent chain, the idea 
is to produce a Markov process {X : \ with the property that z is the equilibrium 
distribution. It can be shown (see [53, 61, 65]) that with rather mild assumptions 
(irreducibility, aperiodicity), the limit (24) holds, due to the law of large numbers. 

In applications to imaging, the computational burden associated with MCMC 
methods has become proverbial and is often presented as the main obstacle to the 
use of Bayesian method in imaging. It is easy to imagine that sampling random 
variable with hundreds of thousands of components will require a large amount of 
computer resources and that collecting and storing a large number of images will 
require much more time than estimating a single one. On the other hand, since 
an ensemble of images from a distribution carries a lot of additional information 
which cannot be included in single-point estimates, it seems unreasonable to rate 
methods simply according to computational speed. That said, since collecting a 
well-mixed, representative sample poses several challenges, in the description of the 
Gibbs sampling and Metropolis—Hastings algorithms, we will point out references 
to variants which can improve the independence and mixing of the ensemble; see 
[30-32]. 

In its first prominent appearance in the imaging arena [26], the Gibbs sampler 
was presented as part of a stochastic relaxation algorithm to efficiently compute 
MAP estimates. The systematic or fully conditional Gibbs sampler algorithm 
proceeds as follows [61]. 

Let m(x) be a probability density defined on R%, denoted by a(x) = 
m(x1,...,Xn), x € R% to underline that it is the joint density of the components 
of X. Furthermore, denote by a(x; | x_j;) the conditional density of the jth 
component x; given all the other components, collected in the vector x_; € RN-!, 
Let x! be the initial element of the Markov chain. Assuming that we are at a point 
x! in the chain, we need a rule stating how to proceed to the next point x'*!, ie., 
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we need to describe the updating method of proceeding from the current element x! 
to x'*!, This is done by updating sequentially each component as follows. 
Fully conditional Gibbs sampling update: Given x', compute the next element 


x'*! by the following algorithm: 
i+1 i). 
draw x} from Zz (x1 [aeae 
i+1 i+1 Vi i). 
draw x, from 1 (x2 | x} eee 9 
i+1 i+] Jit] vi i). 
draw x, from 1 (x3 |. 4 ee se 


draw xit' from x (xv | xi4/). 


In imaging applications, this Gibbs sampler may be impractical because of the large 
number of components of the random variable to be updated to generate a new 
element of the chain. In addition, if some of the components are correlated, updating 
them independently may slow down the chain to explore the full support of the 
distribution, due to slow movementat each step. The correlation among components 
can be addressed by updating blocks of correlated components together, although 
this will imply that the draws must be from multivariate instead of univariate 
conditional densities. 

It follows naturally from the updating scheme that the speed at which the chain 
will reach equilibrium is strongly dependent on how the system of coordinate axes 
relates to the most prominent correlation directions. A modification of the Gibbs 
sampler that can ameliorate the problems caused by correlated components performs 
a linear transformation of the random variable using correlation information. 
Without going into details, we refer to [48, 58,61] for different variants of Gibbs 
sampler. 

The strategy behind the Metropolis—Hastings samplers is to generate a chain 
with the target density as equilibrium distribution by constructing at each step the 
transition probability function from the current X' = x to next realization of X't! 
in the chain in the following way. Given an initial transition probability function 
q(x, x’) with X' = x, x’ drawn from q(x, x’) is a proposal for the value of X‘*!. 
Upon acceptance of X'*! = x’, which occurs with probability a(x, x’), defined by 


m(x')q(x', x) 


a(x,x’) = min ren 


a] , a(x)gq(x,x') > 0. 


We add it to the chain; otherwise, we reject the proposed value and we set X'*! = 
x. In the latter case, the chain did not move and the value x is replicated in the chain. 
The transition probability p(x, x’) of the Markov chain thus defined is 


P(x, x") = q(x, x')a(x, x’), 
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while the probability to stay put is 


1-f q(x, y)a(x, y)dy. 


This construction guarantees that the transition probability satisfies the detailed 
balance equation a(x) p(x, x’) = x(x’) p(x’, x), from which it follows that, for 
reasonable choices of the function g, m(x) is the equilibrium distribution of the 
chain. 

This algorithm is particularly convenient when the target distribution (x) is a 
posterior. In fact, since the only way in which z enters is via the ratio of its values at 
two points, it is sufficient to compute the density modulo a proportionality constant, 
which is how we usually define the posterior. Specific variants of the MH algorithm 
correspond to different choices of g(x,x’); in the original formulation [51], a 
symmetric proposal, for example, a random walk, was used, so that g(x,x’) = 
q(x’, x), implying that 


a(x, x") = min{z2(x’)/z(x), 1}, 


while the general formulation above is due to Hastings [35]. An overview of the 
different possible choices for g(x, x’) can be found in [65]. 

A number of hybrid sampling schemes which combine different chains or use 
MH variants to draw from the conditional densities inside Gibbs samplers have 
been proposed in the literature; see [48, 61] and references therein. Since the design 
of efficient MCMC samplers must address the specific characteristics of the target 
distribution, it is to be expected that as the use of densities becomes more pervasive 
in imaging, new hybrid MCMC scheme will be proposed. 

The convergence of Monte Carlo integration based on MCMC methods is a key 
factor in deciding when to stop sampling. This is particularly pertinent in imaging 
applications, where the calculations needed for additions of a point to the chain 
may be quite time consuming. Due to the lack of a systematic way of translating 
theoretical convergence results of MCMC chains [7,65] into pragmatic stopping 
rules, in practice, the issue is reduced to monitoring the behavior of the already 
collected sample. 

As already pointed out, MCMC algorithms are not sampling independently from 
the posterior. When computing sample-based estimates for the posterior mean and 
covariance, 


. 1 : A 1 ho > oy. po wk 
XcmM = . pa > Tom=- a (x! = Sem) (x! = fu)’. 


A crucial question is how accurately these estimates approximate the posterior mean 
and covariance. The answer depends on the sample size n and the sampling strategy 
itself. Ideally, if the sample vectors x/ are realizations of independent identically 
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distributed random variables, the approximations converge with the asymptotic rate 
1/./n, in agreement with the central limit theorem. In practice, however, the MCMC 
sampling produces sample points that are mutually correlated, and the convergence 
is slower. 

The convergence of the chain can be investigated using the autocovariance 
function (ACF) of the sample [27, 64]. Assume that we are primarily interested 
in estimating a real-valued function f : RY — R of the unknown, and we 
have generated an MCMC sample or a realization 10" Shs x} of a stationary 
stochastic process {X tyibagee a The random variables X/ are equally distributed, 
their distribution being the posterior distribution (x) of a random variable X. The 
estimation of the mean quantity f(X) can be done by calculating 


1 n ; 
i= — 7 fe"), 
j=l 


while the theoretical mean of f(X) is 


w= ELF} = f foymeas. 


Each sample yields a slightly different value for (1, which is itself a realization of 
the random variable F' defined as 


1 n ; 
== sO) 
j=l 


The problem is now how to estimate the variance of F', which gives us an 
indication of how well the computed realization approximates the mean. The 
identical distribution of the random variables X/ implies that 


E{F} = -— SE {X)= 
ae =| EVO} 
=p 
while the variance of F’, which we want to estimate starting from the available 
realization of the by stochastic process, is 


var(F’) = Bir =r. 


To this end, we need to introduce some definitions and notations. 
We define the autocovariance function of the stochastic process f(X/) with lag 
k > Otobe 


Clk) = EE P(X) SIT} = 
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which, if the process is stationary, is independent of 7. The normalized ACF is 
defined as 


C(k) 
c(k) = CO’ 


The ACF can be estimated from an available realization as follows 


n—k 
C0) = YF Fo) — 2. 25) 


j=l 
It follows from the definition of F that 
fee 
ELF} => DUELS SAY}. 
ij=l 


Let us now focus on the random matrix Mmeaned Me _,- The formula above 
takes its expectation and subsequently computes the average of its entries. By 
stationarity, the expectation is a symmetric Toeplitz matrix; hence, its diagonal 
entries are all equal to 


E{f(X) F(X} = CO) +L’, 


while the kth subdiagonal entries are all equal to 


EL f(x) f(X'*)) = CH) + pw’. 


This observation provides us with a simple way to perform the summation by 
accounting for the elements along the diagonals, leading to the formula 


1 n-1 
E{F}= = (new +25 \(n- new) +B, 
k=1 
from which it follows that the variance of F is 
n-1 k 
var(F) = (co +2>° (1 = =) cw). 
k=1 


If we assume that the ACF is negligible when k > no, for some no significantly 
smaller than the sample size n, we may use the approximation 


var(F’) = (co +2 3 cw) = O, T, 


k=1 
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where 


no 


t=14+2) c(k). (26) 


k=l 
If we account fully for all contributions, 


n—-1 


k 
t=1+42 (1 = *) c(k), (27) 


which is the Cesaro mean of the normalized ACFs or low-pass filtered mean with 
the triangular filter. The quantity t is called the integrated autocorrelation time 
(IACT) and can be interpreted as the time that it takes for our MCMC to produce an 
independent sample. If the convergence rate for independence samplers is 1/./n, 
the convergence rate for the MCMC sampler is 1/,/n/t. If the variables X; are 
independent, then t = 1, and the result is exactly what we would expect from the 
central limit theorem, because in this case, C(0) = n var( f(X)). 

The estimate of t requires an estimate for the normalized ACF, which can be 
obtained with the formula (25), and a value for mo to use in formula (26). In the 
choice of ng, it is important to remember that C (kK) is a realization of a random 
sequence C(k), which in practice contains noise. Some practical rules for choosing 
No are suggested in [27]. 

In [27], it is shown that since the sequence 


y(k) = c(2k) +c(2k +1), k=0,1,2,... 


is strictly positive, strictly decreasing, and strictly convex, that is, 


HE) >0, HkEN<yh), YE+N <5 +7 +2), 
when the sample-based estimated sequence, 
p(k) = €(2k) + €(Q2k +1), k=0,1,2,... 
fails to be so, this is an indication that the contribution is predominantly coming 
from noise; hence, it is wise to stop summing the terms to estimate t. Geyer 


proposes three initial sequence estimators, in the following order: 


1. Initial positive sequence estimator (IPSE): Choose no to be the largest integer for 
which the sequence remains positive, 


No = Mpse = max{k | y(k) > O}. 
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2. Initial monotone sequence estimator (IMSE): Choose ng to be the largest integer 
for which the sequence remains positive and monotone, 


No = Nose = maxt{k | y(k) > 0, y(k) < y(k — D}. 


3. Initial convex sequence estimator (ICSE): Choose no to be the largest integer for 
which the sequence remains positive, monotone, and convex, 


eee f | vk) > 0, y(R) < ¥E—D, ye 1) 
1 
< 50k) + yik—2ph 


From the proof in [27], it is obvious that also the sequence {c(k)} itself must be 
positive and decreasing. Therefore, to find ng for IPSE or IMSE, there is no need 
for passing to the sequence {y(k)}. As for ICSE, again from the proof in the cited 
article, it is also clear that the sequence 


n(k) = c(2k +1) +c(2k +2), k=0,1,2,... 


too, is positive, monotonous, and convex. Therefore, to check the condition for 

ICSE, it might be advisable to form both sequences {y(k)} and {n(k)} and set nicsg 

equal to the maximum index for which both y(k) and n(k) remain strictly convex. 
Summarizing a practical rule, using for instance, the IMSE, to compute T is: 


1. Estimate the ACF sequence c (k) from the sample by formula (25) and normalize 
it by C (0) to obtain @(k). 

2. Find no equal to the largest integer for which the sequence ¢(0), ¢(1),...,€(0) 
remains positive and strictly decreasing. Notice that the computation of ACFs 
can be stopped when such an no is reached. 

3. Calculate the estimate for the [ACT rt, 


no k no 
ratt20)(1-F)eW) = 142. et (28) 


Notice that if 7 is not much larger than 10, the sample is too small. 
The accuracy of the approximation of jz by jf is often expressed, with some 
degree of imprecision, by writing an estimate 


1/2 
rae (:) 
nN 


with the 95 % belief. This interpretation is based on the fact that, with a probability 
of about 95 %, the values of a Gaussian random variable are within +2 STD from 
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the mean. Such an approximate claim is justified when n is large, in which case the 
random variable F' is asymptotically Gaussian by the central limit theorem. 


Statistical Approach: What Is the Gain? 


Statistical methods are often pitted against deterministic ones, and the true gain of 
the approach is sometimes lost, especially if the statistical methods are used only to 
produce single estimates. Indeed, it is not uncommon that the statistical framework 
is seen simply as an alternative way of explaining regularization. Another criticism 
of statistical methods concerns the computation times. While there is no doubt 
that computing a posterior mean using MCMC methods is more computationally 
intensive than resorting to optimization-based estimators, it is also obvious that 
a comparison in these terms does not make much sense, since a sample contains 
enormously more information of the underlying distribution than an estimate of its 
mode. 

To emphasize what there is to be gained when using the statistical approach, 
we consider some algorithms that have been found useful and are based on the 
interpretation images as random variables. 


Beyond the Traditional Concept of Noise 
The range of interpretation of the concept of noise in imaging is usually very 
restricted, almost exclusively referring to uncertainties in observed data due to 
exogenous sources. In the context of deterministic regularization, the noise model 
is almost always additive, in agreement with the paradigm that only acknowledges 
noise as the difference between a “true” and “noisy” data, giving no consideration to 
its statistical properties. Already the proper noise modeling of counting data clearly 
demonstrates the shortcomings of such models. The Bayesian — or subjective — use 
of probability as an expression of uncertainty allows to extend the concept of noise 
to encompass a much richer terrain of phenomena, including shortcomings in the 
forward model, prior, or noise statistics itself. 

To demonstrate the possibilities of the Bayesian modeling, consider an example 
where it is assumed that a forward model with additive noise, 


y = F(x) +e. (29) 


which describes, to the best of our knowledge, as completely as possible, the 
interdependency of the data y and the unknown. We refer to it as the detailed model. 
Here, the noise e is thought to be exogenous, and its statistical properties are known. 

Assume further that the detailed model is computationally too complex to be 
used with the imaging algorithms and the application at hand for one or several 
of the following reasons. The dimensionality of the image x may be too high 
for the model to be practical; the model may contain details such as boundary 
conditions that need to be simplified in practice; the deblurring kernel may be non- 
separable, while in practice, a fast algorithm for separable kernels may exist. To 
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accommodate these difficulties, a simpler model is constructed. Let z be possibly a 
simpler representation of x, obtained, for example, via a projection to a coarser grid, 
and let f denote the corresponding forward map. It is a common procedure to write 
a simplified model of the form 


y=f@re, (30) 


which, however, may not explain the data as well as the detailed model (29). To 
properly account for the errors added by the model reduction, we should write 
instead 


y=F(x)+e= fi +[F(x)-— f@] +e 
= f(@+ex,z) +e, e(x,z) = F(x) — f(2), (31) 


where the term é(x, z) is referred to as modeling error. 

In the framework of deterministic imaging, modeling errors pose unsurmountable 
problems because they depend on both the unknown image x and its reduced 
counterpart z. A common way to address errors coming from model reduction is 
to artificially increase the variance of the noise included in the reduced model until 
it masks the modeling error. Such an approach introduces a statistical structure in the 
noise that does not correspond to the modeling error and may easily waste several 
orders of magnitude of the accuracy of the data. On the other hand, neglecting the 
error introduced by model reduction may lead to overly optimistic estimates of the 
performance of algorithms. The very questionable procedure of testing algorithms 
with data simulated with the same forward map used for the inversion is referred to 
as inverse crime [42]. Inverse criminals, who tacitly assume that e(x, z) = 0, should 
not be surprised if the unrealistically good results obtained from simulated data are 
not robust when using real data. 

While modeling error often is neglected also in the statistical framework, its 
statistical properties can be described in terms of the prior. Consider the stochastic 
extension of ¢(x, z), 


E =&(X,Z), 


where X and Z are the stochastic extensions of x and z, respectively. Since, unlike 
an exogenous noise term, the modeling error is not independent of the unknowns Z 
and X, the likelihood and the prior cannot be described separately, but instead must 
be specified together. 

To illustrate how ubiquitous modeling error is, consider the following example. 

Boundary clutter and image truncation: Consider a denoising/deblurring exam- 
ple of the type encountered in astronomy, microscopy, and image processing. Let 
u : R* — R be a continuous two-dimensional model of a scenery that is recorded 
through an out-of-focus device. The noiseless model for the continuous problem is 
a convolution integral, 
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vir) = [. a(r—s)u(s)ds, r e€R’, 


the convolution kernel a(r —s) describing the point spread of the device. We assume 
that r +> a(r) decays rapidly enough to justify an approximation as a compactly 
supported function. 

Let O C R? define a bounded field of view. We consider the following imaging 
problem: Given a noisy version of the blurred image v over the field of view Q, 
estimate the underlying image u over the field of view Q. 

Assume that a sufficiently fine discretization of Q into N pixels is given, and 
denote by r; € QO the center of the ith pixel. Assume further that the point spread 
function a is negligibly small outside a disc D of radius 56 > 0. By selecting an 
extended field of view Q' such that 


Q0+D={seR |s=rt+r,reQ, reD\cQ, 


we may restrict the domain of integration in the definition of the convolution integral 
v(r;) = i! a(r; — s)u(s)ds ® i) a(r; — s)u(s)ds. 
R2 Q’ 


After discretizing Q’ into N’ pixels p; with center points s;, N of which are within 
the field of view, coinciding with R’ we can restate the problem in the form 


N’ 
v(s;) & [i —s)u(s)ds & a |p; la(si — s;)u(s;) 


j=l 


= ajju(s;), aij = |pjlasi — Sj), 1 < I < N. 


After accounting for the contribution of exogenous noise at each recorded pixel, we 
arrive at the complete discrete model 


y=Axt+e, NER, (32) 


where x; = u(s;) and y; represent the noisy observation of v(s;). If the pixelization 
is fine enough, we may consider this model to be a good approximation of the 
continuous problem. 

A word of caution is in order when using this model, because the right-hand side 
depends not only on pixels within the field of view, where we want to estimate the 
underlying image, but also on pixels in the frame C = Q’\Q around it. The vector 
x is therefore partitioned into two vectors, where the first one, denoted by z € RY, 
contains values in the pixels within the field of view, and the second one, ¢ € RK, 
K = N’— N, consists of values of pixels in the frame. After suitably rearranging 
the indices, we may write x in the form 


Statistical Methods in Imaging 1385 


z] RY 
x= Ee x, 
c] R* 


and, after partitioning the matrix A’ accordingly, 
A’ = [AB] ¢ RX*" x RX**, 
we can rewrite the model (32) in the form 
y=Az+B6+e=Azt+erte, 


where the modeling errors are collected in second term ¢, which we will refer to as 
boundary clutter. It is well known that ignoring the contribution to the recorded 
image coming from and beyond the boundary may cause severe artifacts in the 
estimation of the image x within the field of view. In a determinist framework, 
the boundary clutter term is often compensated for by extending the image outside 
the field of view in a manner believed to be closest to the actual image behavior. 
Periodic extension or extensions obtained by reflecting the image symmetrically or 
antisymmetrically are quite popular in the literature, because they will significantly 
simplify the computations; details on such an approach can be found, for example, 
in [21]. 
Consider a Gaussian prior and a Gaussian likelihood, 


X~ N(0, YT), Ew N (0, noise) > 


and partition the prior covariance matrix according to the partitioning of x, 


ie be | . Tre RY*’, Py =P) e RY*®, Ty e REX, 
Ty; Tx 


The covariance matrix of the total noise term, which also includes the boundary 
clutter E, is 


E \(é + E) (E + E)' = BIB" + “noise =r 
and the cross covariance of the image within the field of view and the noise is 
C=E{zZ(£+£)'\ =rpB". 


The posterior distribution of the vector Z conditioned on Y = y now follows from 
(17) and (18). The posterior mean is 
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ZCM = (TuA + TB") Caw + BI2,B" + Sa). J; 
and the posterior covariance is 
= T 
Dpost = Pi — (PuA + T12B") (APH AT + BP22B! + noise) (MuA+TpB") . 


A computationally efficient and robust algorithm for computing the conditional 
mean is proposed in [13]. For further applications of the modeling error approach in 
imaging, see [1,38, 47]. 


Sparsity and Hypermodels 
The problem of reconstructing sparse images or more generally images that can 
be represented as sparse linear combinations of prescribed basis images using data 
consisting of few measurements has recently received a lot of attention and has 
become a central issue in compressed sensing [11]. Bayesian hypermodels provide 
a very natural framework for deriving algorithms for sparse reconstruction. 
Consider a linear model with additive Gaussian noise, the likelihood being 
given by (12) and a conditionally Gaussian prior (13) with hyperparameter 6. 
As explained in section “Adding Layers: Hierarchical Models,” if we select the 
hyperprior Zhyper(@) in such a way that it favors solutions with variances ©; close to 
zero except for only few outliers, the overall prior for (X, ©) will be biased toward 
sparse solutions. Two hyperpriors well suited for sparse solutions are the gamma 
and the inverse gamma hyperpriors. For the sake of definiteness, consider the inverse 
gamma hyperprior with mutually independent components, 


pe 6 6 
TMhyper(O;) = 97 f ‘exp (-3) = exp (-? —(k+ 1) 1080, ) F 
J j 


Then the posterior distribution for the pair (X, ©) is of the form 


N 
1 1 
(x, | y) exp | —5(y —Ax)TE"'(y — Ax) — 5x"Dp!x — 9 VG) 


j=l 


where 


6 
VO) = os + (i + >) log6;, Dg = diag(@) ¢ RN**. 
j 


An estimate for (X, ©) can be found by maximizing z(x, 0 | y) with respect to the 
pair (x, 0) using, for example, a quasi-Newton optimization scheme. Alternatively, 
the following two algorithms that make use of the special form of the expression 
above can also be used. 
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In the articles [66,67] on Bayesian machine learning, the starting point is the 
observation that the posterior density x b> 2(x, 6 | y) is Gaussian and therefore it 
is possible to integrate it explicitly with respect to x. It can be shown, after some 
tedious but straightforward algebraic manipulations, that the marginal posterior 
distribution is 


nly) =f m(x.8 | yds 


1/2 N io - 
) exp | — Vj) + 55"Ma' 5 |, 


1 
7 Exon = 
j=l 
where 
Mp =ATEA+D;51, §#=ATET!y. 
The most probable estimate or the maximum evidence estimator 6 of © is, by 


definition, the maximizer of the above marginal, or equivalently, the maximizer of 
its logarithm, 


N 
1 a 
L(8) = —5 log (det(My)) — ) | V(8;) + 5 5™My'y 
j=l 
which must satisfy 
ba =0, 1<j<WM 
0 ee 


It turns out that, although the computation of the determinant may in general be a 
challenge, its derivatives can be expressed in a formally simple form. To this end 
separate the element depending on 0; from D;' , writing 


where e; is the jth coordinate unit vector, 0’ is the vector 6 with the jth element 
replaced by a zero and “+” denotes the pseudo-inverse. Then 


1 1 
Mo = Alo IA+ Di + mahal = Ma + mahal (33) 
J J 


1 = 
= Me, (! + 74) y q= My/e;. 
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It follows from the properties of the determinant that 
= ee _ qj 
det (Mo) = det{ 1+ 9,78: det (Mg) = {1+ a. det (Mg) , 
J J 


where gj = eyq. After expressing the inverse of Mo in the expression of L(0) via 
the Sherman—Morrison—Woodbury formula [28] as 


My! = Mg — >——qq', 
j 
we find that the function L(@) can be written as 


1 qj 1 (q'3)? 
L(@) = =1 1+)-Vv¢,; a 
(@) ston (1+ 3) VWON+ 35 44; 


+terms that are independent of 6;. 


The computation of the derivative of L(@) with respect to 6; and its zeros is now 
straightforward, although not without challenges because reevaluation of the vector 
q may potentially be expensive. For details, we refer to the article [67]. 

After having found an estimate 6 , an estimate for X can be obtained by observing 
that the conditional density 2(x | y, 6) is Gaussian, 


x 1 1 is 
w(x | y,@) « exp (-30 = Ax) ay Agye 370s) 


and an estimate for x is obtained by solving in the least squares sense the linear 


system 
yA yol/2 
Poe |= : a (34) 
6 


In imaging applications, this is a large-scale linear problem and typically, iterative 
solvers need to be employed [59]. 

A different approach leading to a fast algorithm of estimating the MAP estimate 
(x, @)map was suggested in [15]. The idea is to maximize the posterior distribution 
using an alternating iteration: Starting with an initial value 6 = 6', € = 1, the 
iteration proceeds as follows: 


1. Find x‘*! that maximizes x +> L(x, 6°) = log (x(x, 0° | y)). 
2. Update 0‘*! by maximizing 6 +> L(x‘t!, 6) = log (x(x°*!,@ | y)). 
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The efficiency of this algorithm is based on the fact that for 9 = 6° fixed, the 
maximization of L(x, 0°) in the first step is tantamount to minimizing the quadratic 
expression 


Dae Deal 

SIEM ANP + = [D507] 

2 2 
the non-quadratic part being independent of x. Thus, step 1 only requires an 
(approximate) linear least squares solution of the system similar to (34). On the 
other hand, when x = x°t! is fixed, the minimizer of the second step is found as a 


zero of the gradient of the function L(x‘t+!, @) with respect to 9. This step, too, is 
straightforward, since the component equations are mutually independent, 


) e+ = 1 / e4i\? 1 3\ 1 
70 y= —(5(x ) +4 et k+5) a =o 


leading to the explicit updating formula 


1 2 
£1 £1 
0; = 3 ((x; ) + 26) . 


For details and performance of the method in image applications, we refer to [15]. 


5 Conclusion 


This chapter gives an overview of statistical methods in imaging. Acknowledging 
that it would be impossible to give a comprehensive review of all statistical methods 
in imaging in a chapter, we have put the emphasis on the Bayesian approach, 
while making repeated forays in the frequentists’ field. These editorial choices are 
reflected in the list of references, which only covers a portion of the large body of 
literature published on the topic. The use of statistical methods in subproblems of 
imaging science is much wider than presented here, extending, for example, from 
image segmentation to feature extraction, interpretation of functional MRI signals, 
and radar imaging. 
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1 Introduction 


The desire to learn from examples is as old as mankind but has reached a new 
dimension with the invention of computers. This paper concentrates on learning 
by support vector machines (SVMs) which meanwhile deliver state-of-the-art 
performance in many real-world applications. However, it should be mentioned at 
the beginning that there exist many alternatives to SVMs ranging from classical 
k-nearest neighbor methods over trees and neural networks to other kernel-based 
methods. Overviews can be found, e.g., in the books of Mitchell [73], Hastie et al. 
[47], Duda et al. [34], and Bishop [9]. 

SVMs are a new-generation learning system based on various components 
Including: 


¢ Statistical learning theory, 

¢ Optimization theory (duality concept), 

¢ Reproducing kernel Hilbert spaces (RKHSs), 
e Efficient numerical algorithms. 


This synthesis and their excellent performance in practice make SVM-like learning 
attractive for researchers from various fields. A non-exhaustive list of SVM applica- 
tions includes text categorization (see Joachims [53] and Leopold and Kinderman 
[64]), handwritten character recognition (see LeCun et al. [62]), texture and image 
classification (see Chapelle et al. [23]), protein homology detection (see Jaakkola 
and Haussler [51]), gene expression (see Brown et al. [17]), medical diagnostics 
(see Strauss et al. [96]), and pedestrian and face detection, (see Osuna et al. [77] 
and Viola and Jones [110]). There exist various benchmark data sets for testing and 
comparing new learning algorithms and a good collection of books and tutorials on 
SVMs as those of Vapnik [105], Burges, [19], Cristianini and Shawe-Taylor [27], 
Herbrich [48], Schélkopf and Smola [87], and Steinwart and Christmann [93]. The 
first and latter ones contain a mathematically more rigerous treatment of statistical 
learning aspects. So-called least squares SVMs are handled in the book of Suykens 
et al. [98], and SVMs from the approximation theoretic point of view are considered 
in the book of Cucker and Zhou [29]. 

Let ¥ C R¢,Y C R*, where for simplicity only d = lis considered, and 
Z i= Xx Y. The aim of the following sections is to learn a target function ¥ > Y 
from given training samples Z := {(x;, y1),.--,(%m,¥m)} C Z. A distinction is 
made between classification and regression tasks. In classification Y is a discrete 
set, in general as large as the number of classes the samples belong to. Here binary 
classification with just two labels in VY was most extensively studied. An example 
where binary classification is useful is SPAM detection. Another example in medical 
diagnostics is given in Fig. 1. Here it should be mentioned that in many practical 
applications, the original input variables are pre-processed to transform them into 
a new useful space which is often easier to handle but preserves the necessary 
discriminatory information. This process is also known as feature extraction. 
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Fig. 1 Examples of physiological (SR) and pathological (VT) electrical heart activity curves 
measured by an implanted cardioverter-defibrillator. For the classification of such signals, see 
Strauss and Steidl [95] 


In contrast to classification, regression aims at approximating the “whole” real- 
valued function from some function values, so that Y is not countable here. The 
above examples, as all problems considered in this paper, are from the area of 
supervised learning. This means that all input vectors come along with their 
corresponding target function values (labeled data). In contrast, semi-supervised 
learning makes use of labeled and unlabeled Data, and in unsupervised learning 
labeled data are not available, so that one can only exploit the input vectors x;. The 
latter methods can be applied for example to discover groups of similar exemplars 
within the data (clustering), to determine the distribution of the data within the input 
space (density estimation), or to perform projections of data from high-dimensional 
spaces to lower-dimensional spaces. There are also learning models which involve 
more complex interactions between the learner and the environment. An example 
is reinforcement learning which is concerned with the problem of finding suitable 
actions in a given situation in order to maximize the reward. In contrast to supervised 
learning, reinforcement learning does not start from given optimal (labeled) outputs 
but must instead find them by a process of trial and error. For reinforcement learning, 
the reader may consult the book of Sutton and Barto [97]. 

Learning models can also differ in the way in which the training data are 
generated and presented to the learner. For example, a distinction can be made 
between batch learning, where all the data are given at the start of the training 
phase, and online learning, where the learner receives one example at a time and 
updates the hypothesis function in response to each new example. 

This paper is organized as follows: An overview of the historical background is 
given in Sect. 2. Section 3 contains an introduction into classical SVM methods. It 
starts with linear methods for (binary) support vector classification and regression 
and considers also linear least squares classification/regression. Then the kernel trick 
is explained and used to transfer the linear models into so-called feature spaces 
which results in nonlinear learning methods. Some other models related to SVM as 
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well as multi-class classification and multitask learning are addressed at the end of 
the section. Section 4 provides some mathematical background concerning RKHSs 
and quadratic optimization. The last subsection sketches very briefly some results in 
statistical learning theory. Numerical methods to make the classical SVMs efficient 
in particular for large data sets are presented in Sect.5. The paper ends with some 
conclusions in Sect. 6. 


2 Historical Background 


Modern learning theory has a long and interesting history going back as far as 
Gauss and Legendre but got its enormous impetus from the advent of computing 
machines. In the 1930s revolutionary changes took place in understanding the 
principles of inductive inference from a philosophical perspective, e.g., by Popper, 
and from the point of view of statistical theory, e.g., by Kolmogorov, Glivenko, 
and Cantelli, and applied statistics, e.g., by Fisher. A good overview over the 
leading ideas and developments in this time can be found in the comments and 
bibliographical remarks of Vapnik’s book, Vapnik [105]. The starting point of 
statistical learning theory which considers the task of minimizing a risk functional 
based on empirical data dates back to the 1960s. Support vector machines, including 
their RKHS interpretation, were only discovered in the 1990s and led to an explosion 
in applications and theoretical analysis. 

Let us start with the problem of linear regression which is much older than linear 
classification. The method of least squares was first published by Legendre [63]. 
It was considered as a statistical procedure by Gauss [43], who claimed, to the 
annoyance of Legendre but in accordance with most historians, to have applied 
this method since 1795. The original least squares approach finds for given points 
x; € R@ and corresponding y; € R,i = 1,...,mahyperplane f(x) = (w,x) +b 
having minimal least squares distance from the points (x;, y;): 


Die, x7) +b = yi)? > min. (1) 


i=1 


This leads to the solution of a linear system of equations which can be ill conditioned 
or possess several solutions. Therefore, regularized versions were introduced later. 
The linear least squares approach is optimal in the case of linear targets corrupted 
by Gaussian noise. Sometimes it is useful to find a linear function which does not 
minimize the least squares error, but, for example, the £;-error 


m 
Do l(w. x) +b = yi] > min 
i=l Ww, 


which is more robust against outliers. This model with the constraint that the sum 
of the errors is equal to zero was already studied by Laplace in 1799; see Laplace 


Supervised Learning by Support Vector Machines 1397 


12 12 
11 11 
10 10 
9 9 
8 8 
7 7 
6 6 
7 15 20 25 30 35 40 45 : 15 20 25 30 35 40 45 


12 


15 20 25 30 35 40 45 


Fig. 2. Linear approximation with respect to the £2-, £;- and £49-norm of the error (left to right). 
The €; approximation is more robust against outliers while the £.9-norm takes them better into 
account 


[61]. Another popular choice is the £.9-error 


which better incorporates outliers. In contrast to the least squares method, the 
solutions of the £;- and £.o-problems cannot be computed via linear systems of 
equations but require to solve linear optimization problems. Figure 2 shows a one- 
dimensional example, where data are approximated by lines with minimal £-, £- 
and £49 error norm, respectively. For more information on (regularized) least squares 
problems, the reader may consult, e.g., the books of Golub and Van Loan [45] and 
of Bjérck [10]. 

Regularized least squares methods which penalize the quadratic weight ||w||* 
as in section “Linear Least Squares Classification and Regression” were examined 
under the name ridge regression by Hoerl and Kennard [49]. This method can be 
considered as a special case of the regularization theory for ill-posed problems 
developed by Tikhonov and Arsenin [102]. Others than the least squares loss 
function like the €-insensitive loss were brought into play by Vapnik [105]. This loss 
function enforces a sparse representation of the weights in terms of so-called support 
vectors which are (small) subsets of the training samples {x; :i = 1,...,m)}. 
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The simplest form of classification is binary classification, where one has just 
to separate between two classes. Linear hyperplanes H(w,b) separating points, 
also called linear discriminants or perceptrons, were already studied by Fisher [41] 
and became interesting for neural network researchers in the early 1960s. One of 
the first algorithms that constructs a separating hyperplane for linearly separable 
points was Rosenblatt’s perceptron; see Rosenblatt [84]. It is an iterative online 
and mistake-driven algorithm which starts with an initial weight guess w for the 
hyperplane and adapts the weight at each time a training point is misclassified 
by the current weight. If the data are linearly separable, the procedure converges, 
and the number of mistakes (number of updates of w) does not exceed (2R/y)’, 
where R := min;=}.....m ||x;|| and y is the smallest distance between a training point 
and the hyperplane. For linearly separable points, there exist various hyperplanes 
separating them. 

An optimal hyperplane for linearly separable points in the sense that the minimal 
distance of the points from the plane becomes maximal was constructed as so-called 
generalized portrait algorithm by Vapnik and Lerner [108]. This learning method 
is also known as linear hard margin support vector classifier. The method was 
generalized to nonseparable points by Cortes and Vapnik [26] which leads to soft 
margin classifiers. Finally, the step from linear to nonlinear classifiers via feature 
maps was taken by Boser et al. [12]. Their idea to combine a linear algorithm with a 
kernel approach inspired the further examination of specific kernels for applications. 

However, the theory of kernels and their applications is older than SVMs. 
Aronzajn [5], systematically developed the theory of RHKSs in the 1940s though 
it was discovered that many results were independently obtained by Povzner [83]. 
The work of Parzen [78] brought the RKHS to the fore in statistical problems; see 
also Kailath [54]. Kernels in pattern recognition were already applied by Aizerman 
et al. [1]. Empirical risk minimization over RKHSs was considered by Wahba 
[112] in connection with splines and by Poggio and Giriosi [81] in relation with 
neural networks. Schdlkopf et al. [89] realized that the kernel trick works not 
only for SVMs but for many other methods as principal component analysis in 
unsupervised learning. 

The invention of SVMs has led to a gigantic amount of developments in learning 
theory and practice. The size of this paper would be not enough to list the references 
on this topic. Beyond various applications, also advanced generalization results, 
suitable choices of kernels, efficient numerical methods in particular for large data 
sets, relations to other sparse representation methods, multi-class classification, and 
multitask learning were addressed. The reader will find some references in the 
corresponding sections. 


3 Mathematical Modeling and Applications 


Linear Learning 


This section starts with linear classification and regression which provide the easiest 
algorithms to understand some of the main building blocks that appear also in the 
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more sophisticated nonlinear support vector machines. Moreover, concerning the 
classification task, this seems to be the best approach to explain its geometrical back- 
ground. The simplest function to feed a classifier with or to use as an approximation 
of some unknown function in regression tasks is a linear (multivariate) function 


fWH=70) = We, 2eX CR. (2) 
Often it is combined with some appropriate real number J, i.e., one considers the 
linear polynomial f(x) + b = (w,x) + b. In the context of learning, w is called 
weight vector and b offset, intercept or bias. 
Linear Support Vector Classification 
Let us consider binary classification first and postpone multi-class classification 
to section “Multi-class Classification and Multitask Learning.” As binary classifier 
F = Fy,: X — {-1, l}, one can use 

F(x) := sgn( f(x) + b) = sgn((w, x) + b) 
with the agreement that sgn(0) := 0. The hyperplane 
H(w, b) := {x : (w,x) + b = 0} 


has the normal vector w/'||w||, and the distance of a point X € R@ to the hyperplane 


is given by 
Kier’) aa 
as XxX ——= 
[|| [|| 
see Fig.3 left. In particular, |b|/||w]|| is the distance of the hyperplane from the 
origin. 
The training set Z consists of two classes labeled by +1 with indices J, := 
{i : y; = I} and /_ := {i : y; = —1}. The training set is said to be separable 


by the hyperplane H(w, b) if (w,x;) + b > Ofori € [+4 and (w,x;) + b < 0 for 
iél_,ie., 


yi ((w, x;) +b) > 0. 


The points in Z are called (linearly) separable if there exists a hyperplane separating 
them. In this case, their distance from a separating hyperplane is given by 


(ier) tan) | 
yi | (—,x1)+—], i=1,...,m. 
I|w| I|w| 


The smallest distance of a point from the hyperplane 


y:= min y; (x) + ia) (3) 
i=l... \|w| I | 
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class 1 


H 


Fig. 3 Left: Hyperplane H with normal w/||w]| and distance |b|/||w|] from the origin. The 
distance of the point x from the hyperplane is the value A fulfilling (w, x — Aw/||wl|) + b = 0, 
ie., A = ((w,x) + b)/||w]]. Right: Linearly separable training set together with a separating 
hyperplane and the corresponding margin y 


is called margin. Figure 3, right, shows a separating hyperplane of two classes 
together with its margin. Of course for a separable training set, there may exist 
various separating hyperplanes. One way to ensure a unique solution is to pose 
additional requirements on the hyperplane in form of minimizing a cost functional. 


Hard Margin Classifier One obvious way is to choose those separating hyper- 
plane which has the maximal distance from the data, i.e., a maximal margin. The 
corresponding classifiers are called maximal margin classifiers or hard margin 
classifiers. The hyperplane and the corresponding half-spaces do not chance if the 
defining vectors is rescaled to (c w,c b), c > 0. The so-called generalized portrait 
algorithm of Vapnik and Lerner [108], constructs a hyperplane that maximizes y 
under the constraint ||w|| = 1. The same hyperplane can be obtained as follows: 
by (3), it holds 


so that one can use the scaling 


yiwl=1 2 y= — 


Now y becomes maximal if and only if ||w|| becomes minimal, and the scaling 
means that y; ((w,x;) +b) > 1 for alli = 1,...,m. Therefore, the hard 
margin classifier aims to find parameters w and b solving the following quadratic 
optimization problem with linear constraints: 
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Linear SV hard margin classification (Primal problem) 


S\lwll2 > min subjectto y; ((w,x;) +5) >1,i=1,...,m. 


If the training data are linearly separable, the problem has a unique solution. A brief 
introduction into quadratic programming methods is given in section “Quadratic 
Optimization.” To transform the problem into its dual form, consider the Lagrangian 


Li, b,a) = sibel? + Sr (1 —y; ((w,x;) +b)), a > 0. 


i=1 


Since 
ieee =0 ¢$ = Dans (4) 
a = Day = 0 () 
the Lagrangian can be rewritten as 
L(wb,a) = 2 19 wiaiyja, ix) + oa (6) 


i=1 j=l i=l 


and the dual problem becomes 


Linear SV hard margin classification (Dual problem) 


5 3 Y yay je (Xi,Xj;) — > a > min subjectto 7", yia; = 0, 
i=lj= i= 


a >0,i=1,...,m 


Note that the dual maximization problem has been rewritten into a minimization 


problem by using that max@ = min ae Let 1,, denote the vector with m 
coefficients 1, a := (a; )"_,, y := (w)lL,, Y := diag(y;)?L, and 
K:= ((n,2/))Mer- a) 


Note that K is symmetric and positive semi-definite. The dual problem can be 
rewritten in matrix-vector form as 


Linear SV hard margin classification (Dual problem) 


5a°YKYoa — (1,@) > min subjectto (y,a) =0, a> 0. 
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Let a* be the minimizer of this dual problem. The intercept b does not appear in 
the dual problem, and one has to determine its optimal value in another way. By the 
Kuhn-Tucker conditions, the equations 


a; (yi ((w*, x1) + b*) — 1) =0, i=l,...,m 
hold true, so that a* > 0 is only possible for those training data with y;((w*, x;) + 
b*) = 1. These are exactly the (few) points having margin distance y from the 
hyperplane H(w*, b*). Define 
Ts := fi: a* > 0}, S t= {x :i € Is}. (8) 
The vectors from S are called support vectors. In general |S| < m and by (4) the 


optimal weight w* and the optimal function f,,* have a sparse representation in 
terms of the support vectors 


w= ~ iy Feo= >. oF yi (xi, x). (9) 
ié€ls i€ls 
Moreover, 
b* = yi — (w*, x;) = ji - Fue OG), 1€é I; (10) 


and hence, using (5), 


lw" |? = Sl aty:, D0 at y; (x1.x)) = Do of y: foe) 


i€ls JEls 1€lgs 
* * * 
=) ott") => 
i€ls iéls 


so that 


y= 1/|iwll = (doar. 


1€ls 


Soft Margin Classifier If the training data are not linearly separable which is the 
case in most applications, the hard margin classifier is not applicable. The extension 
of hard margin classifiers to the nonseparable case was done by Cortes and Vapnik 
[26] by bringing additional slack variables and a a parameter C > 0 into the 
constrained model: 


Linear SV soft margin classification (Primal problem) 


Siw? +C & > mm subjectto yy; ((w,x;) +b) > 1-&, 
i=l w.b, 


& >0,i =1,...,m. 
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For C = oo, this is again the hard margin classifier model. As before, the margin 
is defined as y = 1/||w*||, where w* is the solution of the above problem. If the 
slack variable fulfills 0 < &* < 1, then x; is correctly classified, and in the case 
yi ((w*, x;) + b*) = 1—&* the distance of x; from the hyperplane is y — &*/||w* ||. 
If 1 < &*, then one has a misclassification. By penalizing the sum of the slack 
variables, one tries to keep them small. 

The above constrained minimization model can be rewritten as an unconstrained 
one by using a margin-based loss function. A function L : {—1, 1} x R = [0, ow) is 
called margin based if there exists a representing function / : R — [0, 00) such that 


L(y,t) = I(t). 


In soft margin classification the appropriate choice of a loss function is the hinge 
loss function 1, determined by 


I(x) := max{0, 1 — x}. 


Then the unconstrained primal problem reads 


Linear SV soft margin classification (Primal problem) 


g lbw? + CY La Qi, (w. xi) +b) > min. 
j=1 w, 


The Lagrangian of the linear constrained problem has the form 


m m m 


1 
L(w, b,&,0) = sllw|? + C > 1 + Ya: (1 & — yi ((w, xi) +b) — DT BiG. 


i=l i=1 i=l 


where a;, 6; > 0. Partial differentiation of the Lagrangian with respect to w and b 
results in (4), (5) and with respect to € in 


aL 
—=C lyn —-a— = Uz 
rg a—p 0 


Using these relations, the Lagrangian can be rewritten in the same form as in (6), 
and the dual problem becomes in matrix-vector form 


Linear SV soft margin classification (Dual problem) 


5a°YKYo —(1m,@) subjectto (y,a)=0,0<a<C. 


Let w* be the minimizer of the dual problem. Then the optimal weight w* and f,« 
are again given by (9) and depend only on the few support vectors defined by (8). 
By the Kuhn-Tucker conditions, the equations 


a (yi ((w*,x;) +b*)-—14+ &*) =0 and 
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Bret = (Cafe =0, i=1,...,m 


hold true. For 0 < a < C, it follows that &* = 0 and y;((w*, x;) + b*) = lie, 
the points x; have margin distance y = 1/||w*|| from H(w*, b*). Moreover, 


b* = y; —(w*, xi), i € Is = {i : 0<a*<C}. (11) 


For a* = C, one concludes that y;((w*, x;) + b*) = 1 — &*, ie., x; has distance 
y — &*/||w*|| from the optimal hyperplane. 


Linear Support Vector Regression 
Of course one can also approximate unknown functions by linear (multivariate) 


polynomials of the form (2). 


Hard Margin Regression The model for linear hard margin regression is given by 


Linear SV hard margin regression (Primal problem) 


Swill? > min,» subject to (w,x;) +b— y; < €, 
—(w,x;)-bt+y; < € i=l,...,m. 


The constraints make sure that the test data y; lie within an € distance from the value 
F(x) + 6 of the approximating linear polynomial. The Lagrangian reads 


” 1 en 
L(w,b, &*,a*, B*) = sll? + dla; ((w,x:) +b-yi - ©) 


i=l 
+ lat (-(w,x;) -b + yi 6), 
i=1 


where a > 0. Setting partial derivatives to zero leads to 


ab w+ >> (ap - at) x; =0 S$ we Yd) (at - a7) x, (12) 


dw i=1 i=1 
OL m 
= Liat a7) = 0 (3) 


the Lagrangian can be written as 


m m m m 


L(w, b,&*,a~, B )=-5 Do Vaiaylar.xy) et tar) +O vice 


i=l j=1 i=1 i=1 
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and the dual problem becomes in matrix-vector form 


Linear SV hard margin regression (Dual problem) 


5(a+ —a7)"K(at —a7) + €(1n,at +07) — (y,at-—a7) > min 


subjectto (ln,a*—a~)=0, a*>0. 


This is a quadratic optimization problem with linear constraints. Let (a+)*, (a~)* 
be the solution of this problem and a* = (a*+)* —(a~)*. Then, by (12), the optimal 
weight and the optimal function have in general a sparse representation in terms of 
the support vectors x;,i € /s, namely, 


w= Pe tux (x) = = a (x;,x), Ips := fi: a* AO}. (14) 


1€ls iél,s 
The corresponding Kuhn-Tucker conditions are 
(a; )* (e — (w*, x;) —b* + y;) = (15) 
(a")* (€ + (w*, x1) + b* — yj) = 
If (a; )* > Oor (a;*)* > 0, then 
b* = y; —(w*,xi) +6, b* = y; —(w*, xi) -€, 
respectively. Since both conditions cannot be fulfilled for the same index, it follows 
that (a-)* (arz*)* = 0 and consequently, either a* = (a;*)* > 0Oora* = 


—(a;-)* < 0. Thus, one can obtain the intercept by 


b* = y; —(w*,x;)-—¢, ie Ts. (16) 


Soft Margin Regression Relaxing the constraints in the hard margin model leads 
to the following linear soft margin regression problem with C > 0: 


Linear SV soft margin regression (Primal problem) 
zllw? + C Yi, (& + &)> min, , .4 subject to (w.x;) +b-y, Se+&, 
—(w,x;) -b+y;<e+ gt, 

ee 20. 


SF 


For C = o, this recovers the linear hard margin regression problem. The above 
constrained model can be rewritten as an unconstrained one by using a distance- 
based loss function. A function L : Y x R — [0, 00) is called distance based if 
there exists a representing function / : R > [0, co) such that 


L(y,t) =l(y -1). 
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Fig. 4 Vapnik’s €-insensitive loss function for ¢ = 2 (left). Example of linear SV soft margin 
regression (right) 


In soft margin regression, the appropriate choice is Vapnik’s €-insensitive loss 
Junction defined by 


1.(x) := max{0, |x| — €}. 


The function /, is depicted in Fig. 4, left. Then the unconstrained primal model reads 


Linear SV soft margin regression (Primal problem) 


S|)? +C >) LQ, (w, xi) +b) > min. 
i=l w, 


The Lagrangian of the constrained problem is given by 


1 m 
L(w,b, 0%, BX) = s\lwl? + C DEF + &) 


i=1 


+ oar ((w, xi) +b - yi -€-&) 


m 


—>o Bret - > Bre, 


i=1 i=1 


where a;- > 0 and pF > 0,7 = 1,...,m. Setting the partial derivatives to zero 
leads to (12), (13) and 
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OL OL 


qe ne ap a0, i ee Se Se 


Using these relations the Lagrangian can be written exactly as in the hard margin 
problem, and the dual problem becomes in matrix-vector form 


Linear SV soft margin regression (Dual problem) 


$(at —a~)"K(at — a7) + €(Im,a* + 07) — (y,at —a7) > min 


subject to (ln,at—a~)=0, O<at,a~ <C 


If (a+)*, (@~)* are the solution of this problem and a* = (at)* — (a7)*, 
then the optimal weight w* and the optimal function f,,« are given by (14). The 
corresponding Kuhn-Tucker conditions are 


(a; )* (e + (&)* — (w*, x;) —b* + yi) = 0, 
(aj*)* (e + (E*)* + (w*, x1) +b* —y;) = 0, 
(C-@f VE) = 0, (C-@/)VE)*=0, i=1...,m. 


If 0 < (@;*)* or 0 < (a;)*, then 


b* = yi —(w*,x)te+é, b* = y; —(w*, xi) —€ —& , 


respectively. Both equations cannot be fulfilled at the same time so that one can 
conclude that either a* = (a+)* > Oor a = —(a;)* < 0. Incase a* = (a)? 


C, this results in the intercept 
b* = yi —(w*,xi)—€, ie Ts. (17) 


Linear Least Squares Classification and Regression 

Instead of the hinge loss function for classification and the €-insensitive loss 
function for regression, other loss functions can be used. Popular margin-based and 
distance-based loss functions are the logistic loss for classification and regression 


fu —yt CP) tet ne 4er! 
[(yt):=Ini+e") and I(y—t):=—In ate ee 
respectively. In contrast to the loss functions in the previous subsections, logistic 
loss functions are differentiable in ¢ so that often standard methods as gradient 
descent methods or Newton (like) methods can be applied for computing the 
minimizers of the corresponding problems. For details, see, e.g., the book of 
Mitchell [73] or of Hastie et al. [47]. Other loss functions for regression are the 
pinball loss, the Huber function, and the p-th power absolute distance loss |y —t|?, 
p > 0. For p = 2, the latter is the least squares loss 


1408 G. Steidl 


lisg(y —t) = (y —1)°. 


Since (y — t)? = (1 — yt)’ for y € {—1, 1}, the least squares loss is also margin 
based, and one can handle least squares classification and regression using just the 
same model with y € {—1,1} for classification and y € R for regression. In the 
unconstrained form, one has to minimize 


Linear LS classification/regression (Primal problem) 


m 
2 . 
S|lwil? + $ ¥ (wx) +b - yi) > min. 
i=|e—O—_———~~—_—_—_—_————— ’, 
Lisq(yi (w.xi) +b) 


This model was published as ridge regression by Hoerl and Kennard [49] and is a 
regularized version of the Gaussian model (1). Therefore, it can be considered as a 
special case of regularization theory introduced by Tikhonov and Arsenin [102]. The 
minimizer can be computed via a linear system of equations. To this end, rewrite the 
unconstrained problem in matrix-vector form 


1 2 Cc T 2 ‘ 
lho? + = [Xtw + bln — y |? > min, 


where 


X11 +++ Xm 
X:= (X1 ... Xm) = 


Xd +++ Xmd 


Setting the gradient (with respect to w and b) to zero, one obtains 


1 t. 
0= —w+ XX'w + bX1,, — Xy, 


Cc 
0O=1,X'w-liytmb & b=y-(w,x), (18) 
where ¥ := 2°", yj and X := + 7", x;. Hence b and w can be obtained by 


solving the linear system of equations 


eer eT) (n) = (255) 
I I = 4 (19) 


Instead of the above problem, one solves in general the “centered” problem 


—— 
ad 


Linear LS classification/regression in centered version (Primal problem) 


m a 
5 [lve]? a ¢ > (iw, %:) + b-— y;)? > min, 
i=l i, 
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where x; := x; —X,i = 1,...,m. The advantage is that t= Om, Where 0,, is the 
vector consisting of m zeros. Thus, (19) with x; instead of x; becomes a separable 
system, and one obtains immediately that b* = y and that w* follows by solving 
the linear system with positive definite, symmetric coefficient matrix 


ae 1 - 
(5x + a!) w= Ry. 


This means that w is just the solution of the centered primal problem without 
intercept. Finally, one can check by the following argument that indeed w* = w*: 


wb 


ae es De apie ee ek ee : 
(w*, b*) = sein + F Dalir, Bi) + b= yi) | 


i=l] 


- ie Co eye 
w* = argming | sim + > Dw, &) +3- wrt 


i=1 


lL. C m - - — 
= argming} SUI +S Ya) +5 ~ (5) — 9 


i=1 


and with (18) on the other hand 


* *\ , : 1 2 - 2 
(w b ) = argmin,, 7, | 5 Il + 2 2 (wm) F b — yi) | , 
. 1 : C m 7 7 . 
w* = argmin, 4 5llwl? + > > (lw. xi) + 5 — (wk) — yi)? 
i=l 
Note that X"X = K ¢€ R”’”, but this is not the coefficient matrix in (19). When 
turning to nonlinear methods in section “Nonlinear Learning,” it will be essential to 
work with K = X'X instead of XX". This can be achieved by switching to the dual 
setting. In the following, this dual approach is shown although it makes often not 
sense for the linear setting since the size of the matrix K is in general larger than 
those of XX" € R¢@“. First, one reformulates the primal problem into a constrained 


one: 


Linear LS classification/regression (Primal problem) 


m 
sw? +9 Oe = min subjectto (w,xi) +b—y; = §&, i =1,...,m. 
i=l w,b, 


The Lagrangian reads 


1 C m m 
L(w, b, &,0) = Sllwi? + > D8 — Dai (ws) +b yi — 8) 


i=1 i=1 
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and 
mere a =0 S$ we Das 
ee = 0, 
ence te = 0. 


The equality constraint in the primal problem together with the above equalities 
leads to the following linear system of equations to determine the optimal a* and 


b*: 
0) u \(b\_ (0 
(2 K+ =) (") 7 ()). = 


The optimal weight and the corresponding optimal function read 


w* — ret Fur (Xx) = 5 atti: (21) 


i=1 i=] 


In general there is no sparse representation with respect to the vectors x;; see also 
Theorem 8.36 in the book of Steinwart and Christman [93]. Therefore, this method 
is not called a support vector method in this paper. Finally, note that the centered 
approach also helps to avoid the intercept in the dual approach. Since this is no 
longer true when turning to the nonlinear setting, the intercept is kept here. 


Nonlinear Learning 


A linear form of a decision or regression function may not be suitable for the task at 
hand. Figure 5 shows two examples, where a linear classifier is not appropriate. 

A basic idea to handle such problems was proposed by Boser et al. [12] and 
consists of the following two steps which will be further explained in the rest of this 
subsection: 


1. Mapping of the input data X C ¥ into a feature space ®(V) C £2(), where I 
is a countable (possibly finite) index set, by a nonlinear feature map 


®:X > (1). 


2. Application of the linear classification/regression model to the feature set 
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Fig. 6 Linearly nonseparable training data in the original space X C R? (left) and become 
separable in the feature space ®(%), where ®(x1, x2) := (x2, x3) (right) 


(Pr), V1), +++ (Pm), Ym)}- 


This means that instead of a linear function (2), we are searching for a function 
of the form 


F(x) = f(x) = (w, ®(X))en(1) (22) 
now. This nonlinear function on Y becomes linear in the feature space ®(V). 


Figure 6 shows an example of a feature map. In this example, the set {(x;, y;) : 
i = 1,...,5} is not linearly separable while {(®(x;), y;): i = 1,...,5} is linearly 
separable. In practical applications, in contrast to this example, the feature map often 
maps into a higher dimensional, possibly also infinite dimensional space. 

Together with the so-called kernel trick to avoid the direct work with the feature 
map 9, this approach results in the successful support vector machine (SVM). 
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Kernel Trick 

In general, one avoids to work directly with the feature map by dealing with the 
dual problem and applying the so-called kernel trick. For a feature map ®, define a 
kernel K : X x X — R associated with ® by 


K(x, t) = (®(x), PM) pi). (23) 


More precisely, in practice one often follows the opposite way, namely, one starts 
with a suitable kernel which is known to have a representation of the form (23) 
without knowing © explicitly. 

A frequently applied group of kernels are continuous, symmetric, positive (semi-) 
definite kernels like the Gaussian 


K(x,t) =e bore 


These kernels, which are also called Mercer kernels, will be considered in detail 
in section “Reproducing Kernel Hilbert Spaces.” By Mercer’s theorem, it can be 
shown that a Mercer kernel possesses a representation 


K(x.) => Puiwi aw. etek 


jel 


with L2-orthonormal functions w; and positive values A ;, where the right-hand side 
converges uniformly. Note that the existence of such a representation is clear, but in 
general without knowing the functions y; explicitly. The set {g; := i AiW yise 
I} forms an orthonormal basis of a reproducing kernel Hilbert space (RKHS) Hx. 
These spaces are considered in more detail in section “Reproducing Kernel Hilbert 
Spaces.” Then the feature map is defined by 


(x) := (9)(0) ,e) = (37 W009) 


Using the orthonormal basis, one knows that for any f € Hx there exists a unique 
sequence w = wy := (wj) jer € €2(Z) such that 


ef 


F(x) = D> wo; (%) = (w, B(x), and wy = (fg) ate: (24) 


jel 


where the convergence is uniformly. Conversely every sequence w € £2(/) defines 
a function f, lying in 1x by (24). Moreover, Parseval’s equality says that 


Il fellate *= lilac: (25) 


Supervised Learning by Support Vector Machines 1413 


For nonlinear classification and regression purposes, one can follow exactly the 
lines of section “Linear Learning” except that one has to work in ®(V) instead of 
X. Using (22) instead of (2) and 


K := ((®(x), P(x) 00); p= =(KG4p), (26) 


ij=l 


instead of the kernel matrix K := (xi, x Da in (7), the linear models from 
section “Linear Learning” can be rewritten as in the following subsections. Note 
again that K is positive semi-definite. 


Support Vector Classification 
In the following, the linear classifiers are generalized to feature spaces. 


Hard Margin Classifier The hard margin classification model is 


SVM hard margin classification (Primal problem) 


sllleay > min subject toy; ((w, P(x) ) ou) +6) = 1, i =1,...,m. 
Ww, 


Interestingly, if ® is associated with a Mercer kernel K, then f(x) = (w, ®(x))¢,(7) 
lies in the RKHS 7x, and the model can be rewritten using (25) from the point of 
view of RKHS as 


SVM hard margin classification in RKHS (Primal problem) 
1 2 . ; ; 
- > bject t F i b)>1,i=1,...,m. 
al T lla, Jay. Suen (f(xi) +b) = 1, 3 m 


The dual problem reads 


SVM hard margin classification (Dual problem) 


m m 


2 De pz yi yj; (P(x), P(X) OW) — 2% > min subject to > yia; = 0, 


i=1 


a >0,i=1,...,m 


and with K defined by (26) the matrix-vector form of the dual problem looks as 
those for the linear hard margin classifier. 

Let a* be the minimizer of the dual problem. Then, by (9) together with the 
feature space modification, the optimal weight and the function /,,« become 


= Vat yiG(xi). fur (e) = Dat y (O04), 0) ou) = Yat yi K(x. x). 
ié€ls ié€ls ié€ls 

(27) 

Thus, one can compute f,,* knowing only the kernel and not the feature map itself. 

One property of a Mercer kernel used for learning purposes should be that it can 

be simply evaluated at points from VY x %. This is, for example, the case for the 

Gaussian. Finally, using (10) in the feature space, the intercept can be computed by 
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b* = y, — (w*, B(x) ny = Yi — DoF yj/KOxj, xi), TE Ts 
Jéls 


and the margin y = 1/\w* lec by using I" loa = (a*)"YKYa*. 


Soft Margin Classifier The soft margin classification model in the feature space is 


SVM soft margin classification (Primal problem) 


sll +C DY §& =m subject to yj ((w, P(x) +b) = 1- &, 
i=l w,D, 


i=1,...,m 
& = 0, i=1,...,m. 


If ® is associated with a Mercer kernel K, the corresponding unconstrained version 
reads in the RKHS 


SVM soft margin classification in RKHS (Primal problem) 


5 cy La (Qi. f(Xi) +b in. 
si lla, + Xu an (Qi, fi) + b) + = 


With K defined by (26), the matrix-vector form of the dual problem looks as 
those for the linear soft margin classifier. The function f\,« reads as in (27), and, 
using (11), the intercept can be computed by 


b* = v= (w*, ®(x;)) oc) = yi - > a; yj K(x;,%1), i¢€ Is. 


jeéls 


Support Vector Regression 
In the following, the linear regression models are generalized to feature spaces. 


Hard Margin Regression One obtains 


SVM hard margin regression (Primal problem) 
sll —> min,» subject to (w, ®(x;))e~7 tb-—yi <e, 
—(w, ®(x)) eu —b + Yi <€, i=1l,...,m. 


If ® is associated with a Mercer kernel K, then f(x) = (w, ®(x))¢,(7) lies in the 
RKHS Hx, and the model can be rewritten using (25) from the point of view of 
RKHS as 


SVM hard margin regression in RKHS (Primal problem) 


sIlF lau, > minfepy, subjectto f(x;)) +b-yi <e, 
—f(xj)-b+y<¢e, i=1,...,m. 


The dual problem reads in matrix-vector form as the dual problem for the linear 
SV hard margin regression except that we have to use the kernel matrix K 
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defined by (26). Let (wt)*, (w~)* be the solution of this dual problem and a* = 
(a+)* — (a—)*. Then one can compute the optimal function f,* using (14) with the 
corresponding feature space modification as 


fur (x) = YO af K(x, x). (28) 


i€l-s 


One obtains sparse representations in terms of the support vectors. By (16), the 
intercept can be computed by 


b* = yi— D- a*K(xj,mi)—€, iE Is. 


Jélrs 


Soft Margin Regression In the feature space, the soft margin regression model is 


SVM soft margin regression (Primal problem) 
slwlizgy + C DiLiG* +) > min, + st. (w, O@i))Quy +b-vi E+E, 


—(w, ®(x;)) e (1) —b+ yviset+ er 
gre 20. 


wob,é7 


Having a feature map associated with a Mercer kernel K, the corresponding 
unconstrained problem can be written as the following minimization problem in 
the RKHS Hx: 


SVM soft margin regression in RKHS (Primal problem) 


sil f lle +C Lei, fi) +5) > min. 
i=l fEHK 


The dual problem looks as the dual problem for linear SV soft margin regression 
but with kernel (26). From the solution of the dual problem (at)*, (a—)*, one can 
compute the optimal function /,* as in (28) and the optimal intercept using (17) as 


b* = yi- > a K(xj,xj))—-€, ies. 


JElrs 
Figure 7, left, shows an SVM soft margin regression function for the data in Fig. 2. 


Relations to Sparse Approximation in RKHSs, Interpolation by Radial 
Basis Functions, and Kriging 

SVM regression is related to various tasks in approximation theory. Some of them 
will be sketched in the following. 


Relation to Sparse Approximation in RKHSs Let Hx be a RKHS with kernel 
K. Consider the problem of finding for an unknown function f € Hx with given 


FS (xi) = yi,i = 1,...,m an approximating function of the form 
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Fig. 7 SVM regression using the Gaussian with c = 8 for the data in Fig.2. SVM soft margin 
regression curve with C = 0.5 and e« = 0.2 (left). Least squares SVM regression curve with 
Cc =40 


m 


f(x) = Daj K(x, x) € Hx (29) 


i=1 


with only few summands. A starting point would be to minimize the #x-norm of 
the error and to penalize the so-called £9-norm of w given by ||a@||o := |{i : a; 4 0}| 
to enforce sparsity. Unfortunately, the complexity when solving such £9-penalized 
problems increases exponentially with m. One remedy is to replace the £9-norm by 
the €;-norm, i.e., to deal with 


1 i m ; 
SIF) — Dla K (x, xi)llieg + €lleell: > min, (30) 


i=1 


where € > OQ. This problem and its relation to support vector regression were 
considered by Girosi [44] and Evgeniou et al. [37]. Using the relations in RKHS 
from section “Reproducing Kernel Hilbert Spaces,” in particular the reproducing 
property (H2), this problem becomes 


m 


1 1, x 
50'Ka — ) laiyi + sli flliug + lel > min, (31) 


i=l 
where K is defined by (26). With the splitting 


+ or, of > 0, ato> =0,i=1,...,m 


and consequently |a;| = at + a; , the sparse approximation model (30) has finally 
the form of the dual problem of the SVM hard margin regression without intercept: 
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SVM hard margin regression without intercept (Dual problem) 


5(at —a~)"K(at — a7) + €(Im,a* +07) — (y,at-—a7) > amin 
subject to a*~ > 0. 
Note that for « > O the additional constraints a;"a° = O,i = Il,...,m 


are automatically fulfilled by the minimizer since otherwise, the Kuhn-Tucker 
conditions (15) without intercept would imply the contradiction f(x;) = y; +¢« = 


yi — €. 


Relation to the Interpolation by Radial Basis Functions For « = 0, prob- 
lem (30), resp. (31) becomes 


F(a) := 50'Ka -aty > min. 
If K is positive definite, the solution of this problem is given by the solution of 
VF(a) =Ka-—y=0, @& Ka=y 
and the approximating function f reads 


f(%) = (Ky, (K@, x1) Fi). (32) 


This is just the solution of the interpolation problem to find f of the form (29) such 
that f(x;) = y; for alli = 1,...,m. If the kernel K of the positive definite matrix 
arises from a radial basis function k(x) = k(||x — ¢||?), i.e., K(x, t) = k(x — 2B) as, 
e.g., from a Gaussian or an inverse multiquadric described in section “Reproducing 
Kernel Hilbert Spaces,” this interpolation problem is called interpolation by radial 
basis function. 

If the kernel K arises from a conditionally positive definite radial function « of 
order v, e.g., from a thin plate spline, the matrix K is in general not positive semi- 
definite. In this case, it is useful to replace the function f in (29) by 


m n 


f(x) = Dia K(x, x1) + D> Bere, 


i=1 k=1 


where n is the dimension of the polynomial space T1,-1(R?) and {pri k = 
1,...,n} is a basis of T,-;(R%). The additional degrees of freedom in the 
interpolation problem f(x;) = y;,i = 1,...,m are compensated by adding the 
new conditions 


Sai pe(xi) =0, k=1,...,n. 


i=1 
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This leads to the final problem of finding aw := (a@;)7_, and B := (B;)Z_, such that 


r E mn 
(> >) (3) - GE P= (pe(xi))ea1 - (33) 


If the points {x; : i = 1,...,m},m > dim (T1,-;(R4)) are T1,_;(R“)-unisolvent, 
i.e., the zero polynomial is the only polynomial from I1,_;(R®) that vanishes on all 
of them, then the linear system of equations (33) has a unique solution. To verify 
that the coefficient matrix in (33) is indeed invertible, consider the corresponding 
homogeneous system. Then the second system of equations P’a = O means that a 
satisfies (39). Multiplying the first system of equations by a’ gives 0 = a’Ka + 
(P'a)"B = a'Ka. By the definition of conditionally positive definite functions of 
order v, this is only possible if # = 0. But then PB = 0. Since the points {x; :i = 
1,...,m} are T1,_;(R“)-unisolvent, this implies that 6 = 0. 

The interpolation by radial basis functions having (conditionally) positive defi- 
nite kernels was examined, including fast evaluation techniques for the interpolating 
function f, by many authors; for an overview see, e.g., the books of Buhmann 2003 
[18], Wendland [114], and Fasshauer 2007 [39]. 


Relation to Kriging The interpolation results can be derived in another way by 
so-called kriging. Kriging is a group of geostatistical techniques to interpolate 
the unknown value of a random field from observations of its value at nearby 
locations. Based on the pioneering work of Krige [59] on the plotter of the distance- 
weighted average gold grades at the Witwatersrand reef in South Africa, the French 
mathematician Matheron [70] developed its theoretical foundations. Let S(x) 
denote a random field such that the expectation value fulfills E(S(X)) = 0 which is 
the setting in the so-called simple kriging. Let K(x;,x;) := Cov(S(x;), S(x;)) 
and K := (K (xi,X; Vy at? The aim is to approximate the value S(xo) from 
observations S(x;) = yj,i = 1,...,m by the kriging estimator 


§(xo) = » (x0) S (xi) 


i=l 
in such a way that the variance of the error is minimal, i.e., 


Var(S — S) = Var($) + Var(S) — 2Cov(S, $) 
_ a > Wi (x0)@; (X) K(x, x;) 
i=1 j=1 


—2 @; (x0) K (x0, xi) + Var(S) — min. 
dX (x0) K(x0, x7) + Var(S) > min 
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Setting the gradient to zero, the minimizer w* = (oy (Xo), +++, @n, (x0))" is given 
by the solution of the following linear system of equations: 


Ko = (K(x0, Xi));=) - 
In case K is invertible, we get 
S(xo) = (y, K7! (K(x0, x))1) = (Ky, (K(X0, x1))fL1). 


Supposing the same values K(x;, x;) as in the interpolation task, this is exactly the 
same value as f(xo) from the radial basis interpolation problem (32). 


Least Squares Classification and Regression 

Also in the feature space, least squares classification and regression can be treated by 
the same model. So-called Least Squares Support Vector Classifiers were introduced 
by Suykens and Vandevalle [99] while least squares regression was also considered 
within regularization network approaches, e.g., by Evgeniou et al. [37] and Wahba 
[112]. The least squares model in the feature space is 


LS classification/regression in feature space (Primal problem) 


aWllenay + 5 X (iw, P(x))euy +b yi)? > min. 


and becomes in the case that the feature map is related with a Mercer kernel K a 
problem in a RKHS Hx: 


LS classification/regression in RKHS (Primal problem) 


si f lle + $ 2 (fCi)+5-yi) > min. 
i=1 w, 


Setting gradients to zero, one can try to solve this primal problem via a linear system 
of equations (19) with X := (®(x,)... ®(x,)). However, one has to compute with 
XX" here which is only possible if the feature space is finite dimensional. In contrast, 
the dual approach leads to the linear system of equations (20) which involves only 
the kernel matrix K = X™X from (26). Knowing the dual variable a*, the optimal 
function /,,* can be computed using (21) with the feature space modification as 


m m 


fue (x) = Do af (®(x)), 0) D0 a7 K(x, x). 


i=l i=1 


In general there is no sparse representation with respect to the vectors x;. For more 
information on least squares kernel methods, the reader may consult the book of 
Suykens et al. [98]. Figure 7, right, shows a least squares SVM function for the data 
in Fig. 2. 
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Other Models 

There are numerous models related to the classification and regression models 
of the previous subsections. A simple classification model which uses only the 
hinge loss function without penalizing the weight was proposed by Bennet and 
Mangasarian [7]: 


m 


pee (i. (w, X;) +b) => mn, 


i=l ied 
This approach is called robust linear programming (RLP) and requires only linear 
programming methods. Note that the authors weighted the training errors by 1/n+, 
where n+ := [{i;y; = +1}]. The linear SV soft margin classifier adds just the 
penalizer A llwll5 with A = 1/C,0 < C < o to the RLP term which leads to 
quadratic programming. Alternatively, one can add instead the £2-norm the £;-norm 
of the weight as it was done by Bradley and Mangasarian [16]: 


Do Ln (vie (x2) +b) + Allwll: > min. 


i=] 


As in (30), the €; penalizer enforces the sparsity of the solution vector w*. Note that 
the sparsity of w* itself and not a sparse representation of w* as linear combination 
of the support vectors x; is announced here. The £,-penalizer was introduced in 
the statistical context of linear regression in conjunction with the least squares loss 
function by Tibshirani [101] and is called “LASSO” (Least Absolute Shrinkage and 
Selection Operator): 


De bsg (Vi (ws xi) +.B) + Allwll: > min 


i=] 


As emphasized in section “Support Vector Regression,” the £;-norm is more or less 
a replacement for the £9-norm to make problems computable. Other substitutes of 
the £o-norm are possible, e.g., ||w]|e, ~ yl —e!il), » > 0 which gives 


m d 
Lh T> AT Xr 1— ul i ii 
y n i, (w, xi) +) + LI el) — min 


This is a nonconvex model and was proposed by Bradley and Mangasarian [16] and 
by Weston et al. [115] as FSV (Features Selection concaVe). Numerical solution 
methods via successive linearization algorithms and difference of convex functions 
algorithms by Tao and An [100] were applied. 

Further, one can couple several penalizers and generalize the models to the 
feature space to obtain nonlinear classifiers as it was done, e.g., by Neumann et 
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Fig. 8 From left to right: (i) Sample CT slice from a three-dimensional scan of the data set with 
contours of bladder and prostate. (ii) A zoom within the region of interest shows that the organs 
are very difficult to distinguish visually. (iii) Manual classification by an expert as “accepted truth.” 
(iv) A classification result: The images are filtered by a three-dimensional steerable pyramid filter 
bank with 16 angular orientations and four decomposition levels. Then local histograms are built 
for the filter responses with ten bins per channel. Including the original grey values, this results in 
650 features per image voxel which are used for classification by the “¢-€9-SV” machine 


al. [75]. Figure 8 shows an example for the binary classification of specific organs 
in CT scans with a so-called “€2-€9-SV” machine taken from the above paper, where 
more information can be found. In particular, a 77 kernel was used here. 


Multi-class Classification and Multitask Learning 

So far only binary classification was considered. Assume now that one wants to learn 
K > 2 classes. Figure 9 shows a typical example of 12 classes for the classification 
of Mammals; see Amit et al. [2]. 

Some attempts to extend the binary case to multi-classes were achieved by adding 
constraints for every class; see Weston and Watkins [116] and Vapnik [105]. In case 
of many classes, this approach often results in quadratic problems which are hard 
to solve and difficult to store. In the following, two general approaches to handle 
multiple classes are presented, namely, with: 


(i) Vector-valued binary class labeling, 
(ii) K class labeling. 


The dominating approach for solving multi-class problems using SVMs is based on 
reducing a single multi-class problem to multiple binary problems. For instance, 
a common method is to build a set of binary classifiers, where each classifier 
distinguishes between one of the labels and the rest. This so-called one-versus- 
all classifier cannot capture correlations between different classes since it breaks 
the multi-class problem into independent binary problems. More general, one can 
assign to each class a vector-valued binary class label (y,..., y)" € {-1, 1} 
and use a classifier based on 


F(x) := (sen((w, x) + b)) 1 : 
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Black+Rat 


Deer+Mouse 


Fig. 9 Classification of mammal images: 12 from 72 classes of animals which were used for 
classification in Amit et al. [2]. Typically these classes share common characteristics as in the 
different rows above (deers, canines, felines, rodents), e.g., the texture or shape 


For example, in the one-versus-all method, the classes can be labeled by {(—1 + 
2.4) :r =1,...,K}, ie, « = K, and the assignment of x to a class can be 
made according to the shortest Hamming distance of F(x) from these class labels. 
In the one-versus-all example, there was k = K. More sophisticated methods use 
values k > K and error-correcting output codes as Dietterich and Bakiri [32]. Note 
that 2* different labels are in general possible with binary vectors of length « which 
is an upper bound for the number of classes that could be learned. In the learning 
process, one can obtain w) € R” and b“ € R by solving, e.g., 


m 


1 K K 
= (k) 2 dw) x. (k) i 
2 Dw | + 206 DOL (rw i) +b) > min (34) 


where L is some loss function. Note that this problem can be decoupled with respect 
tok. Let W := (w ...w“)) € R@ be the weight matrix. Then the first sum in (34) 
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coincides with the squared Frobenius norm of W defined by 


K d 
Wiz = 2 wy. 


k=1i=1 


Let us consider the second labeling approach. Here one assumes that each class label 


is an integer from Y := {1,..., K}. As before, one aims to learn weight vectors 
wk =1,...,K (the intercept is neglected for simplicity here). The classifier is 
given by 


Fy (x) := argmax (w”, x). 


A training sample (x;, y;) is correctly classified by this classifier if 
(w8),x;) > (Ww, x) +1, We =1,...,K, k # yj. 


Without adding | at the left-hand side of the inequality, correct classification is still 
attained if there is strong inequality for k # y;. This motivates to learn the weight 
vectors by solving the minimization problem 


|W > min subject to (w9), x:) + 8y,4¢ — (Ww, x;) = 1, 


Vk =1,...,K andi =1,...,m. 
After introducing slack variables to relax the constraints, one gets 


5 |W lz + C0 E> maneabyect to(w??), x;) + by, 6 — (w®, x) >1-&, 
Vk =1,...,K andi =1,...,m. 


This can be rewritten as the following unconstrained problem: 


aa On) () 
sIWIR + Doh (iw Xi) + max((w Xi) — By, bk) | > min . 


i=1 


In this functional the learning tasks are coupled in the loss function. 

In general the aim of multitask learning is to learn data that are common across 
multiple related supervised learning tasks, i.e., to facilitate “cooperative” learning. 
Recently, multitask learning has received attention in various applications; see 
the paper of Caruana [21]. Learning of vector-valued functions in RKHSs was 
considered, e.g., by Micchelli and Pontil [72]. Inspired by the “sparseness” models 
in section “Other Models” which focus on the sparsity of the weight vector w, one 
can ask for similar approaches for a weight matrix W. As a counterpart of the 
£o-norm of a weight vector, there can serve a low rank of the weight matrix. But 
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as in £9-penalized minimization problems, such problems are computationally not 
manageable. A remedy is to replace the low-rank condition by demanding a small 
trace norm or nuclear norm of W defined by 


W]e = do o;, 
J 


where o; are the singular values of W. Then a minimization problem to learn the 
weight matrix reads, for example, as 


1 K m 
5 lll + C SL (yi, (w™, x;)) .—— 7 


k=1i=1 


where L is mainly the least squares loss function. Such models were considered by 
Amit et al. [2], Obozinski et al. [76], and Pong et al. [82]. Other approaches use the 
norm 


d 
Wa 


j=l 


which favors a small number of nonzero rows in W instead of the trace norm; 
see Argyriou et al. [4] and Obozinski et al. [76]. Another interesting model was 
proposed by Argyriou et al. [4] and learns in addition to a weight matrix an 
orthogonal matrix U € O by minimizing 


m 


K 
|Wlo1 + C Der (yi. (w®, Ux;)). > yan. 


The numerical solution of multitask problems which are convex but non-smooth 
requires sophisticated techniques. The trace norm minimization problem can be, for 
example, reformulated as a semi-definite program (SDP), and then existing SDP 
solvers can be used as long as the size of the problem is moderate; see the papers 
of Fazel et al. [40] and Srebro et al. [91]. A smooth, but nonconvex reformulation 
of the problem and a subsequent solution by a conjugate gradient or alternating 
minimization method was proposed, e.g., by Weimer et al. [113]. Accelerated 
proximal gradient methods (multistep methods) and Bregman iterative methods 
were applied in the papers of Lu et al. [66], Ma et al. [67], Cai et al. [20], and 
Toh and Yun [103]. A new primal-dual reformulation of the problem in conjunction 
with a gradient projection method to solve the reduced dual problem was given by 
Pong et al. [82]. 
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Applications of SVMs 

SVMs have been applied to many real-world problems. Some applications were 
already sketched in the previous subsections. Very often SVMs are used in 
connection with other techniques, in particular feature extraction/selection methods 
to specify the input domain. 

A non-exhaustive list of SVM applications includes text categorization (see 
Joachims [53] and Leopold and Kinderman [64]), hand-written character recog- 
nition (see LeCun et al. [62]), texture and image classification (see Chapelle et al. 
[23]), protein homology detection (see Jaakkola and Haussler [51]), gene expression 
(see Brown et al. [17]), medical diagnostics (see Strauss et al. [96]), and pedestrian 
and face detection (see Osuna et al. [77], Viola and Jones [110]). 

This subsection describes only two applications of SVM classification and shows 
how the necessary design choices can be made. In particular, one has to choose an 
appropriate SVM kernel for the given application. Default options are Gaussians or 
polynomial kernels, and the corresponding SVMs often already outperform other 
classification methods. Even for such parameterized families of kernels, one has to 
specify the parameters like the standard deviation of the Gaussian or the degree of 
the polynomial. In the Gaussian case, a good choice of the standard deviation in the 
classification problem is the distance between closest points within different classes. 
In the absence of reliable criteria, one could use a validation set or cross-validation 
to determine useful parameters. Various applications require more elaborate kernels 
which implicitly describe the feature space. 


Handwritten Digit Recognition The problem of handwritten digit recognition 
was the first real-world task on which SVMs were successfully tested. The results 
are reported in detail in the book of Vapnik [105]. This SVM application was so 
interesting because other algorithms incorporating prior knowledge on the USPS 
database have been designed. The fact that SVMs perform better than these specific 
systems without using prior detailed information is remarkable; see [62]. 

Different SVM models have been tested on two databases: 


¢ United States Postal Service (USPS): 7,291 training and 2,007 test patterns of the 
numbers 0,...,9, represented by 16 x 16 gray level matrices; see Fig. 10. 

¢ National Institute of Standard and Technology (NIST): 60,000 training and 
10,000 test patterns, represented by 20 x 20 gray-level matrices. 


In the following, the results for the USPS database are considered. 
For constructing the decision rules, SVMs with polynomial and Gaussian kernels 


were used: 


K(x,t) = ((x,1)/256)", K(x, t) = eT #1125607), 
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seas ou aie uees CEE MQLE Id ORR ALA ML WZZIE. 
database; see [105] O24 23BQEIOLOSSE2ODAICOSPS32 
DS4OZOS SIL E7II7TSISALSIZLS 
OYO311.35378$224900029893 
AZISIZIOIACIYRTOOZ SOLOS TS 
PS ITOZIITIIAIZULIZELOZS LL, 
bOSRDOISSIOILTIATIINZ9OS 7.1. 
995.2.1,02992786 2538701 7231S27¥0 
OXI LELAIGLIGIGIIALG LN GGA 
AMZESTEARZOZZAG YS ZI AIA IES, 
WAALELBIALYROTOILASZAL TE 
R48 2074200 LYe2r- VOUS ZS +250) 
AS &SSLALRLELASLY LAOS YL 3 ch 
KIN GIPQTIAAHZ AAI AYOLZA SOS. 
DELZAUATSZOAYAS LYS SO142.0 1. 
SIIQLEZEZIS7YOIS3YO-030S+ 
U4 LOSO¥CHLI47993596 139902 
222244 LSOSSCEASOZEIOOOSS 
QStOb4IISEd INE LLANE SOR ¢. 
Ghd GSEASRLOVSEZIIIZELZ IL 


The overall machine consists of 10 classifiers, each one separating one class from 
the rest (one-versus-all classifier). Then the 10-class classification was done by 
choosing the class with the largest output number. 

All types of SVMs demonstrated approximately the same performance shown in 
the following tables, cf. [105]. The tables contain the parameters for the hard margin 
machines, the corresponding performance, and the average (over one classifier) 
number of support vectors. Moreover, it was observed that the different types of 
SVMs use approximately the same set of support vectors. 


Deareen CE Ca 
Error 8.9 4.7 4.0 4.2 4.5 4.5 
Number of SV 282 237 274 321 374 422 


Results for SVM classification with polynomial kernels 


o 4.0 1.5 0.3 0.25 0.2 0.1 
Error 5.3 4.9 4.2 4.3 4.5 4.6 
Number of SV 266 237 274 321 374 422 


Results for SVM classification with Gaussian kernels 
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Fig. 11 Labeled USPS examples of training errors for the SVM with second degree polynomial 
kernel; see [105] 


Finally, it is worth to mention that the training data are not linearly separable; 
~5 % of these data were misclassified by a linear learning machine. For a degree 
2 polynomial kernel, only the 4 examples in Fig.11 were misclassified. For 
polynomials of degree 3, the training data are separable. The number of support 
vectors increases only slowly with the degree of the polynomial. 


Color Image Recognition Image recognition is another area where SVMs were 
successfully applied. Chapelle et al. [23] have reported their SVM classification 
results for color image recognition. The database was a subset (Corel14) of the Corel 
Stock Photo Collection consisting of 1,400 photos associated with 14 categories. 
Each category was split into 2/3 for training and 1/3 for testing. Again the one- 
versus-all classifier was applied. 

The images were not used themselves as inputs, but each image was associated 
to its color histogram. Since each color is a point in a three-dimensional vector 
space and the number of bins per color was fixed at 16, the dimension of such a 
histogram (and thus of the feature space) is d = 16°. Note that low-level features 
like histograms have the advantage that they are invariant with respect to many 
operations and allow the comparison of images of different sizes. Of course, local 
high-level image features like edges are not captured by low-level features. Chapelle 
and co-workers [23] have used both the RGB (red, green, blue) and the HSV/HSB 
(hue, saturation, value/hue, saturation, brightness) histogram representation. Note 
that HSV arranges the geometry of RGB in an attempt to be more perceptually 
relevant. As kernels they have used 


K(x, t) = eo dist(x.)/07 


where dist denotes a measure of similarity in the feature space which has to be 
determined. For histograms, the y? function 


z —t;) 
dist(x,t) := yee (i ie 
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is accepted as an appropriate distance measure. It is not clear if the corresponding 
kernel is a Mercer kernel. For the distances dist,(x,f) := ||x — |b, p = 1,2 this 
is the case. 

As can be seen in the following table, the SVM with the y? and the £, distance 
perform similarly and significantly better than the SVM with the squared (2 
distance. Therefore, the Gaussian kernel is not the best choice here. RGB- and HSV- 
based methods perform similarly. 


Linear Degree 2 poly ye £y Gaussian 
RGB 42.1 33.6 28.8 14.7 14.7 
HSV 36.3 35.3 30.5 14.5 14.7 


Error rates (percent) using different SVM kernels 


For comparision, Chapelle and co-workers conducted some experiments of color 
image histogram (HSV-based) classifications with the K-nearest neighbor algorithm 
with y? and £7. Here K = 1 gives the best result presented in the following table: 


qe Ly 
26.5 47.7 


Error rates (percent) with k-nearest neighbor algorithm 


The x?-based SVM is roughly twice as good as the y?-based k-nearest neighbor 
technique. 


4 Survey of Mathematical Analysis of Methods 
Reproducing Kernel Hilbert Spaces 
General theory For simplicity, let % C R¢ be a compact set throughout this 


subsection. Moreover, only spaces of real-valued functions are considered. Let 
C(4%) denote the set of continuous functions on V. Together with the norm 


II FIle) = supt] f@)| sx € ¥} 
this becomes a Banach space. Further, let L2(4%) be the Hilbert space of (equiva- 


lence classes) quadratic integrable, real-valued functions on ¥ with inner product 
and norm given by 


(Kadi fi fener. fll = ( : foas) 


Since ¥ is compact, the space C(’) is continuously embedded into L2(4’) which 
means that || f ||2(~) < Cl fIlccay for all f €¢ C(¥). A function K: ¥ x X > R 
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is symmetric, if K(x,t) = K(t,x) for all x,t € 4. With a symmetric function 
K € L,(X x &), one can associate an integral operator Tx : L(V) > L2(4) by 


Tk f(t) := i. K(x, t) f(x) dx. 


This operator is a compact and self-adjoint operator, and K is called its kernel. The 
following spectral theorem holds true for compact, self-adjoint operators, i.e., in 
particular for 7x. 


Theorem 1 (Spectral theorem for compact, self-adjoint operators). Let T be a 
compact, self-adjoint operator on the Hilbert space H. Then there exists a countable 
(possibly finite) orthonormal system {W; : i € I} and a zero sequence (Aj)ier, 
Ai € R\{0} such that 


H = kerT @ span{y; :i € T} 
and 


TE=DAMAW uv; VP EX. (35) 


jel 


The numbers 4; are the nonzero eigenvalues of T and w; are the corresponding 
eigenfunctions. If T is a positive operator, i.e., 


prs i i K(x,t) f(x) f(t)dxdt >0 Vf eH, 


then the values A; are positive. 


Consider the special operator 7x for a symmetric kernel K € L2(¥ x 4X). Using 
the L2-orthonormal eigenfunctions {yj : i € I} of Tx, one can also expand the 
kernel itself as 


K(x,t) = 4 VWiX)W;(O, 


jel 


where the sum converges as those in (35) in general only in L2(X x #). One can 
tighten the statement if K is continuous and symmetric. Then Tx : C(V) > C(4) 
is a compact operator on the Pre-Hilbert spaces C(4’) equipped with the L2-norm 
into itself, and the functions y; are continuous. If f € C(%), then the right-hand 
side in (35) converges absolutely and uniformly. To prove such a convergence result 
also for the kernel expansion, one needs moreover that the operator 7x is positive. 
Unfortunately, it is not true that a positive kernel K implies a positive operator 
Tx. There is another criterion which will be introduced in the following. A matrix 
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K € R”"” is called positive semi-definite if 
a'Ka>0 VaeR"” 


and positive definite if strong inequality holds true for alla 4 0. A symmetric kernel 


K: Xx X => Ris positive (semi)-definite if the matrix K := (K(xi.x; Dy is 
positive (semi)-definite for all finite sets {x1,..., Xm} C 4. Now asymmetric kernel 


K € C(# x #&) is positive semi-definite if and only if the corresponding integral 
operator Tx is positive. 


Theorem 2 (Mercer’s theorem). Let K € C(X x &) be a continuous, symmetric, 
and positive semi-definite function with corresponding integral operator Tx. Then 
K can be expanded into an absolutely and uniformly convergent series in terms 
of Tx’s orthonormal eigenfunctions w; and the associated eigenvalues A; > 0 as 
follows: 


K(x,t) = oAjvj vj (0. (36) 


jel 


Moreover, if K is positive definite, then {y; : i € I} form an orthonormal basis of 
L4(X). 


A continuous, symmetric, positive semi-definite kernel is called a Mercer kernel. 
Mercer kernels are closely related to so-called reproducing kernel Hilbert spaces. 

Let H bea real Hilbert space of functions f : 4 > R.A function K: 4% xX > 
R is called a reproducing kernel of H if 


(Hl) Ky := K(-,tHheH Vted, 
(H2) ft) =(fKikun Vf € Hand Vt €X (Reproducing Property). 


In particular, property (H2) implies for f := )Y/_, a) Kx, and g := DY, Bj Kx, 
that 


m 


(fede = 2 a8) Kei.t), fi => Yo aia; KO. t)) = o"Ka, 
i=1j=1 i=1ij=l 
j j (37) 
where @ = (@),...,@,)"’ and K := (K(x; xy ie If such a kernel exists for H, 
then it is uniquely determined. A Hilbert space which exhibits a reproducing kernel 
is called reproducing kernel Hilbert space (RKHS); write H = Hx to emphasize 
the relation with the kernel. In (x, the set of all finite linear combinations of K,, 


t € &X is dense, i.e., 


Hx = span{K;:t € X}. (38) 
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Moreover, the kernel K of a RKHS must be a symmetric, positive semi-definite 
function; see Wendland [114]. Finally, based on the Riesz representation theorem, 
another characterization of RKHSs can be given. It can be shown that a Hilbert 
space H. is a RKHS if and only if the point evaluation functionals 6, : # —> R 
determined by 5, f := f(x) are continuous on H, i.e., 


If@ISCllflle Vf eH. 
Conversely, by the following theorem, any Mercer kernel gives rise to a RKHS. 


Theorem 3. Let K € C(X x 4&) be a continuous, symmetric, and positive semi- 
definite function. Then there exists a unique Hilbert space Hx of functions on X 
which is a RKHS with kernel K. The space Hx consists of continuous functions on 
X, and the embedding operator tx : Hx(¥) > C(X)( — L(X)) is continuous. 


Proof. 1. First, one constructs a Hilbert space which fulfills (H1) and (H2). By 
(H1), the space Hx has to contains all functions K;, t € ¥ and since the space 
is linear also their finite linear combinations. Therefore, define 


Ho := span{K,:t € XY}. 


According to (37), this space can be equipped with the inner product and 
corresponding norm 


m n 


(f.g)wo = >> oa Bj) K(xi.t;), Shy = "Ko. 


i=1 j=l 


It can easily be checked that this is indeed an inner product. In particular 
Il f llxo = O for some f = >", a; Ky, implies that f(t) = 77_, a; K(t, xi) = 
0 for allt € ¥ by the following argument: set x,,4, := t. By the positive semi- 
definiteness of K is follows for any € € R that 


(a",€) (K(x, x))) Fret ) = a'Ka + 2e Yo ai K(xi.1) de R(t) = 0. 


i=l 
With aKa = || f |lzz) = 0, this can be rewritten as 
€ (2 f(t) + €K(t,t)) = 0. 


Since K is positive semi-definite, it holds K(t,t) => 0. Assume that f(t) < 0. 
Then choosing 0 < «€ < —2f(t)/K(t,t) if K(@t,t) > OandO < eé if 
K(t,t) = 0 leads to a contradiction. Similarly, assuming that f(t) > 0 and 
choosing —2 f(t)/K(t,t) < « < Oif K(t,t) > Oande < Oif K(t,t) = 0 gives 
a contradiction. Thus f(t) = 0. 
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Now one defines 1x to be the completion of Ho with the associated norm. 
This space has the reproducing property (H2) and is therefore a RKHS with 
kernel K. 

2. To prove that Hx is unique, assume that there exists another Hilbert space H of 
functions on ¥ with kernel K. By (H1) and (38), it is clear that Ho is a dense 
subset of H.. By (H2) it follows that (f, g)4, = (fi g)au, forall fg € Ho. Since 
both H and Hx are completions of Ho, the uniqueness follows. 

3. Finally, one concludes by the Schwarz inequality that 


IFO] =A Kral SWF lle Kelle = WF lla VAD 


so that f is continuous since K is continuous. Moreover, || f ||c.v) < Cll F lle, 


with C := max;ex / K(t,t) which means that the embedding cx is continuous. 
oO 


Since the completion of Ho is rather abstract, another characterization of Hx 
based on Mercer’s theorem is useful. Let {y; : i € J} be the L2-orthonormal 
eigenfunctions of 7x with corresponding eigenvalues 4; > O from the Mercer 
theorem. Then it follows by Schwarz’s inequality and Mercer’s theorem for w := 
(wi )ier € £2(/) that 


1/2 
Yo wi Vai)! < Iwlle (x avi) = Iwlle VK@, x) 
ie] ie] 


so that the series }°,., wi V/A; Wi (x) converges absolutely and uniformly for all 
(w; ier € £21). Now another characterization of Hx can be given. 


Corollary 1. Let K ¢ C(X x &) be a continuous, symmetric, and positive semi- 
definite kernel with expansion (36). Then the Hilbert space 
H:= 3 wiv divi: wiier € OU) 
ie] 
with inner product 


(hale = De =o Wale Wyte 
jai 


ie] 


for f = Diep wiv divi and g = DV jer Oj VA; VW; is the RKHS with kernel K, 
ie, H = Hx. The system {g; := JA; Wi : i € I} is an orthonormal basis of H. 
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If K is positive definite, then H can be also characterized by 


H = (f € LX): Mf vy dal? <0} 
Ei 


j=l 


Proof. By the above definition of the inner product, the set {./A;W; : i € 7} is an 
orthonormal basis of H. The second equality in the definition of the inner product 
follows by the orthonormality of the y; in L2. 

It remains to show that K fulfills (H1) and (H2). Concerning (H1), it holds K; = 
Vier VAI Vi (t) VA Vi, and since 


DWaivi(O)? = K(t,1) < 00, 


ie] 


it follows that K, € H. Using the orthonormal basis property, one can conclude 
with respect to (H2) that 


(fda = (wy fis. Vii OVI Wn = Domi VO = £0. 


jel iel iel 
oO 
For more information on RKHS, see the book of Berlinet and Thomas-Agnan [8]. 


Kernels The choice of appropriate kernels for SVMs depend on the application. 
Default options are Gaussians or polynomial kernels which are described together 
with some more examples of Mercer kernels below: 


1. Let ¥ := {x € R¢: ||x|| < R} with radius R > 0. Then the dot product kernels 


ie, [o,) 
K(x,t):= Do aj(x-t)/, a; =0, D0 a;R™ < 00 
j=0 j=l 


are Mercer kernels on ¥. A proof that these kernels are indeed positive semi- 
definite is given in the book of Cucker and Zhou [29]. A special case appears if 
* contains the coordinate vectors e;, 7 = 1,...,d and the kernel is K(x,t) = 
1 + x-t. Note that even in one dimension d = 1, this kernel is not positive 
definite. Here the corresponding RKHS 7x is the space of linear functions and 
{1,x1,..., xa} forms an orthonormal basis of Hx. 

The special dot product K(x,t) := (c+ x-t)",c = 0,n €N, also known as 
polynomial kernel was introduced in statistical learning theory by Vapnik 1998 
[105]. More general dot products were described, e.g., by Smola, Schéllkopf and 
Miiller [89]. See also all-subset kernels and ANOVA kernels in [88]. 
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2. Next, consider translation invariant kernels 
K(x,t):= k(x —-f), 


where « : R¢ — R is a continuous function which has to be even, that is, 
k(—x) = K(x) for all x € R@ to ensure that K is symmetric. Let us see if K 
is a Mercer kernel on R@ and hence on any subset X of R¢. First, one knows 
from Bochner’s theorem that K is positive semi-definite if and only if it is the 
Fourier transform of a finite nonnegative Borel measure on R¢. Let « € L;(R?). 
Then, K is positive definite if and only if « is bounded and its Fourier transform 
is nonnegative and non-vanishing. 

A special example on R (d = 1) is the spline kernel K generated by the 
“hat function” k(x) := max{0,1 — |x|/2}. Its Fourier transform is K(w) = 
2(sinw/w@)? > 0. Multivariate examples of this form can be constructed 
by using, e.g., box splines. Spline kernels and corresponding RKHSs were 
discussed, e.g., by Wahba [112]. 

3. A widely used class of translation invariant kernels are kernels associated with 
radial functions. A function k : R¢ -> R is said to be radial if there exists a 
function k : [0,0o) — R such that x(x) = k(||x||7) for all x € R®. For radial 
kernels define 


K(x,t) := k(||x — tl”). 
A result of Schoenberg [85] says that K is positive semi-definite on R@ if 


and only if the function k is completely monotone on [0, 00). Recall that k is 
completely monotone on (0, 00) if k € C™(0, co) and 


(-1)/k(r) > 0 V1 € No and Yr > 0. 


The function k is called completely monotone on [0, co) if it is in addition in 
C[0, co). 


It holds that K is positive definite if and only if one of the following conditions 
is fulfilled 


(i) k(,/-) is completely monotone on [0, oo) and not constant, 
(ii) There exists a finite nonnegative Borel measure v on [0,00) that is not 
concentrated at zero such that 


k(r) = fae dv(t). 


The proofs of these results on radial kernels are contained, e.g., in the book of 
Wendland [114]. 
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For c > 0, the kernels K arising from the following radial functions x are positive 
definite: 


xl] /e2 : 
K(x) := eT IF/" Gaussian, 


K(x) = (c? + ||x|!?)~°, s > 0 inverse multiquadric, 


where the positive definiteness of the Gaussian follows from (i) and those of the 
inverse multiquadric from (ii) with 


1 pe 2 2 
k(r) = =a! plete t dt. 
P(s) Jo 


Positive definite kernels arising from Wendland’s radial basis functions with com- 
pact support (see Wendland [114]) were applied in SVM classification by Strauss 
and Steid] [95]. 

Finally, the following techniques for creating Mercer kernels are remarkable. 


Theorem 4. Let K; € C(4# x X), j = 1,2 be Mercer kernels and p a polynomial 
with positive coefficients. Then the following functions are also Mercer kernels: 


@) K(x, 1) = Ki(x,t) + Ka(x, 2), 
(ii) K(x, t) := Ki(x, t)Ko(x, 1), 
(iii) K(x,t) — P(Ki (x,t), 

(iv) K(x,t):= ekilxt), 


II 


Beyond the above Mercer kernels other kernels like kernels for text and 
structured data (strings, trees), diffusion kernels on graphs or kernel incorporating 
generative information were used in practice; see the book of Shawe-Taylor and 
Cristianini [88]. 


Conditionally positive semi-definite radial functions. In connection with radial 
basis functions, so-called conditionally positive semi-definite functions k(x) := 
k(||x||?) were applied for approximation tasks. Let T1,_;(R“) denote the space of 
polynomials on R¢ of degree < v. This space has dimension dim(IT,_;(R“)) = 
(“+”). A continuous radial function « : R¢ — R is conditionally positive semi- 
definite of order v if for all m € N, all pairwise distinct points x1,...,%m € R%, 
and all a € R’”\{0} satisfying 


Yo aj p(x) =0 Vp € T,-1(R*) (39) 
i=l 
the relation 


a"Ka>0,  K:= (k(x — xj) 2, 


1436 G. Steidl 


holds true for all a € R’”. If equality is attained only for a = 0, the function x is 
said to be conditionally positive definite of order v. 

The following result is due to Micchelli [71]: For k € C[0,co) N C™(0, co), 
the function k(x) := k(||x||?) is conditionally positive semi-definite of order v if 
and only if (—1)’k™ is completely monotone on (0, 00). If k is not a polynomial 
of degree at most v, then « is conditionally positive definite of order v. 

Using this result, one can show that the following functions are conditionally 
positive definite of order v: 


K(x) = (-1)81(c? + ||x|?)’, s > 0,5 ¢N, v = [fs] multiquadric, 
K(x) = (-DP/7 1 [x]%, s > 0,8 €2N, v = [s/2], 
K(x) = (—1)*t" || x|]* log |x|], k €N, v =k +1 thin plate spline. 


A relation of a combination of thin plate splines and polynomials to the reproducing 
kernels of certain RKHSs can be found in Wahba [112]. 


Quadratic Optimization 


This subsection collects the basic material from optimization theory to understand 
the related parts in the previous Sect. 3, in particular the relation between primal 
and dual problems in quadratic programming. More on this topic can be found 
in any book on optimization theory, e.g., in the books of Mangasarian [68] or 
Spellucci [90]. 

A (nonlinear) optimization problem in R¢ has the general form 


Primal problem (P) 
(x) > min subjectto g(x) <0, h(x) =0 


where 6 : R¢ — R isa real-valued function and g: R? > R",h: Rt? > R? 
are vector-valued functions. In general, only the case p < d is of interest since 
otherwise one is confronted with the solution of a (nonlinear) system of equations. 
The region 


G := {x ER? : g(x) < 0,A(x) = 0}, 


where the objective function @ is defined and where all constraints are satisfied, is 
called feasible region. There are classes of problems (P) which are well examined as 
convex optimization problems and in particular special classes of convex problems, 
namely, linear and quadratic problems. Problem (P) is called convex, if 6 is a convex 
function and G is a convex region. Recall, that x* € G is a local minimizer of 0 
in G if there exists a neighborhood U/(x*) of x* such that @(x*) < O(x) for all 
x € U(x*) NG. For convex problems, any local minimizer x* of @ in G is also a 
global minimizer of @ in G and therefore a solution of the minimization problem. 
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This subsection deals mainly with the following setting which gives rise to a convex 
optimization problem: 


(C1) @ convex and differentiable, 
(C2) g;,i = 1,...,m convex and differentiable, 
(C3) hj, j =1,..., p affine linear. 


Important classes of problems fulfilling (C1)-(C3) are quadratic programs, where 
the objective function is quadratic and the constraints are (affine) linear and linear 
programs, where the objective function is also linear. The constrained optimization 


problems considered in Sect. 3 are of this kind. 
The function L : R¢ x IR" x R? — R defined by 


m P 
L(x, a, B) := @(x) + Yai gi (x) + S> Bj hj (x) 
i=l j=l 
is called the Lagrangian associated with (P), and the coefficients a; and 6; are 


called Lagrange multipliers. Recall that (x*,A*) € Qx ©,Q Cc KR’, E CR’ is 
called a saddle point of a function ® : Q x & > Rif 


O(x*, A) < O(x*,A*) < O(x,A*) V(xX,A)EQx =. 


There is the following close relation between saddle point problems and min-max 
problems: 


Lemma 1. Let ®: Q x & — R. Then the inequality 


max min ®(x, A) < minmax ®(x, A) 
AEE xEQ xEQ LES 


holds true supposed that all extreme points exist. Moreover, in this case, the equality 


max min ®(x,A) = ®(x*, A*) = min max ®(x, A) 
NEE xEQ xEQ AEE 


is fulfilled if and only if (x*,A*) is a saddle point of ®. 


The solution of (P) is related to the saddle points of its associated Lagrangian as 
detailed in the following theorem. 


Theorem 5. /f (x*, (a*, B*)) € R?x (IR! xR?) is a saddle point of the Lagrangian 
associated with the minimization problem (P), i.e., 


L(x*,a, B) < L(x*,a*, B*) < L(x,0*, B*) Vx €R4,V(a,B) € R" xR’, 
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then x* is a solution of (P). Assume that the functions 0, g,h satisfy the conditions 
(C1)-(C3) and that g fulfills in addition the following Slater condition: 


there exists xo € Q such that g(xo) > 0 and h(xo) = 0. 


Then, if x* is a solution of (P), there exist a* € R' and B* € R? such that 
(x*, (a*, B*)) is a saddle point of the associated Lagrangian. 


By the next theorem, the minimizers of (P) can be also described via the following 
conditions on the Lagrangian: there exist x* € G, a* € R” and B* € R? such that 


(KKT1) V,L(x*,a*, B*) =0, 
(KKT2) (a*)'g(x*) =0, a*>0. 


These conditions were independently established by Karush and Kuhn and Tucker 
[60] and are mainly called Kuhn-Tucker conditions. 


Theorem 6. Let 0, g and h fulfill (C1)-(C3). If x* satisfies (KKT1)-(KKT2), then 
x* is a solution of (P). Assume that g fulfills in addition the Slater condition. Then, 
if x* is a solution of (P), it also fulfills (KKT1)-(KKT2). 
If there are only equality constraints in (P), then a solution is determined by 
V,L(x*, B*) =0, h(x*) =0. 
For the rest of this subsection, assume that (C1)—(C3) and the Slater condition 
hold true. Let a solution x* of (P) exist. Then, by Lemma 1 and Theorem 5, there 


exist a* and B* such that 


L(x*,a*, B*) = max min L(x,q, B). 


R7p x 


Therefore, one can try to find x* as follows: for any fixed (a, 8B) € RY x R?, 
compute 


X(a, B) := argmin, L(x, a, B). (40) 
If 6 is uniformly convex, i.e., there exists y > 0 such that 
18(x)+(1—p)O(y) = O(uxt+(l—p) y)+uU—p)y |lx—y ll? Vx,» € RS pw € [0,1], 
then x(a, 6) can be obtained as the unique solution of 


V,L(x,a, B) = 0. 
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This can be substituted into L which results in w(a, B) := L(X(q@, B), a, B) and a* 
and f* are the solution of 


y(a,B) > eg subjectto a> 0. 


This problem, which is called the dual problem of (P) can often be more easily 
solved than the original problem since one has only simple inequality constraints. 
However, this approach is only possible if (40) can easily be solved. Then, finally 
x* = X(a*, B*). 

The objective functions in the primal problems in Sect.3 are not strictly 
convex (and consequently also not uniformly convex) since there does not appear 
the intercept b in these functions. So let us formulate the dual problem with 
w(x,a, B) := L(x,qa, B) as follows: 


Dual problem (D) 
w(x,a,B) > ey subjectto V,L(x,a,8)=0, a> 0. 


The solutions of the primal and dual problem, i.e., their minimum and maximum, 
respectively, coincide according to the following theorem of Wolfe [117]. 


Theorem 7. Let 0, g and h fulfill (C1)-(C3) and the Slater condition. Let x* be 
a minimizer of (P). Then there exist a*, B* such that x*,o*, B* solves the dual 
problem and 


A(x") = w(x", a", B*). 


Duality theory can be handled in a more sophisticated way using tools from 
Perturbation Theory in Convex Analysis; see, e.g., the book of Bonnans and Shapiro 
[11]. Let us briefly mention the general idea. Let v : R” — (—oo, oo] be an extended 
function, where only extended functions ¥ oo are considered in the following. The 
Fenchel conjugate of v is defined by 


v"(a) := sup {(a, x) — v(x)} 
peER” 


and the biconjugate of v by v** := (v*)*. In general, the inequality v**(x) < 
v(x) holds true and becomes an equality if and only if v is convex and lower 
semicontinuous. (Later the inequality is indicated by the fact that one minimizes 
the primal and maximizes the dual problem.) For convex, lower semicontinuous 
functions 6 : R4 — (—oo, co], y : R” > (—oo, 00] and g : R4 > R” one 
considers the primal problems 


(Pu) vu) = int) + (g(x) + wf, 
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(P) v(0) = inf, {0(x) + r(g(@))} 


where u € R” is the “perturbation.” With L(x,a@) := 0(x) + (g(x), qa), the dual 
problem reads 


(D,)v**(u) = sup {(ar.u) — y"(a) + ink L(x,0)}, 
aeR” xeR?@ 


(D) v"*(0) = sup {—y"(@) + inf L(x, a)}. 
aeR” xeER¢@ 
For the special setting with the indicator function 


= __ JO ify <0, 
vO) = ae) oo otherwise 


the primal problem (P) is equivalent to 
A(x) — min subjectto g(x) <0 
and since y* = ip the dual problem (D) becomes 


sup inf L(x,q@) subjectto a> 0. 
aeR™ xER¢ 


Again, if 9 and g are convex and differentiable and 6 is uniformly convex, then the 
unique solution x(a) of V,L(x, a) = 0 is the solution of the infimum problem, and 
the dual problem becomes sup, cpm L(X(@), a) subject toa > 0. 


Results from Generalization Theory 


There exists a huge amount of results on the generalization abilities of statistical 
learning methods and in particular of support vector machines. The following 
subsection can only give a rough impression on the general tasks considered in this 
field from a simplified mathematical point of view that ignores technicalities, e.g., 
the definition of the correct measure and function spaces and what measurable in the 
related context means. Most of the material is borrowed from the book of Steinwart 
and Christmann [93], where the reader can find a sound mathematical treatment of 


the topic. 
To start with, remember that the aim in Sect. 3 was to find a function f : ¥ > R 
from samples Z := {(x;, yj) : i = 1,...,m} such that f(x) is a good prediction 


of y at x for (x, y) € X x Y. Let P denote an unknown probability distribution on 
xX x Y. Then a general assumption is that the data used in training and testing are 
identically independent distributed (iid) according to P. The loss function or cost 
function L : X x ¥xR = [0, co) describes the cost of the discrepancy between the 
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prediction f(x) and the observation y at x. The choice of the loss function depends 
on the specific learning goal. In the models of this paper, the loss functions depend 
on x only via f(x) such that one can simply write L(y, f(x)). In Sect. 3, the hinge 
loss function and the least squares loss function were used for classification tasks. 
Originally, one was interested in the 0/1 classification loss Lo; : Y x R — {0, 1} 
defined by 


0 if y = sgn(t), 
L H:= 
oi (1) 1 otherwise. 
To the loss function, there is associated a risk which is the expected loss of /: 


Rio(f= [Lowy fooyaPony) = ff Ley. fonaPorlsaPr. 
Xxy xJy 
For example, the 0/1 loss function has the risk 


Rig,.p(f) = P (x,y) €¥ XY: senf(x) # y). 


A function f is considered to be “better” the smaller the risk is. Therefore, one is 
interested in the minimal risk or Bayes risk defined by 


R* ,:= inf R “ 41 
L.P Poe LP(Sf) (41) 


where the infimum is taken over all possible (measurable) functions. However, since 
the distribution P is unknown, it is impossible to find a minimizer of Rzp. In 
learning tasks one can exploit finite training sets Z of iid data. A learning method 
on X x Y maps every data set Z € (¥ x VY)" toa function fz : ¥ > R.A 
learning methods should produce for sufficiently large training sets Z nearly optimal 
decision functions fz with high probability. A measurable learning method is called 
L-risk consistent for P if 


lim P”™(Z €(¥ x)": Ri p(fz) < Ri pte) =1 VWe>O 
m—>oo , 


and universally L-risk consistent, if it is L-risk consistent for all distributions P on 
xX x Y. The first learning method that was proved to be universally consistent was 
the so-called nearest neighbor method; see Stone [94]. Many uniformly consistent 
classification and regression methods are presented in the books of Devroye et al. 
[30] and Gyorfi [46]. Consistency does not address the speed of convergence, i.e., 
convergence rates. Unfortunately, the no-free-lunch theorem of Devroye [31], says 
that for every learning method there exists a distribution P for which the learning 
methods cannot produce a “good” decision function in the above sense with an a 
priori fixed speed of convergence. To obtain uniform convergence rates, one has to 
pose additional requirements on P. 
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Instead of the risk one can deal with the empirical risk defined by 


m 


Riz(f) = = YL Gi. $i). 


i=1 


Then the law of large numbers shows that Rz.z(f) becomes a “good” approxi- 
mation of Rr p(f) for a fixed f if m is “large enough.” However finding the 
minimizer of 


joe Riz(f) (42) 


does in general not lead to a good approximation of R7_p. For example, the function 
which classifies all x; € X correctly and is zero elsewhere is a minimizer of the 
above functional (42) but gives in general a poor approximation of the optimal 
decision function according to (41). This is an example of so-called overfitting, 
where the learning method approximates the training data too closely and has 
poor generalization/prediction properties. One common way to cope with this 
phenomenon is to choose a smaller set F¥ of functions, e.g., subsets of continuous 
functions, which should have good approximation properties. In the SVMs treated 
in Sect.3, this set F was a RKHS Hx. Then one considers the empirical risk 
minimization (ERM) 


pos Riz(f). (43) 


Let a minimizer fz of (43) be somehow “computed.” (In this subsection, the 
question of the existence and uniqueness of a minimizer of the various functionals 
is not addressed.) Then one is of course interested in the error R; p( fz) — Rz p- 
Using the infinite-sample counterpart of the ERM 


RF := inf R 
L.P.F a LpP(f) 
this error can be splitted as 


Rip (fz) — Rip = Rip(fz)—Ripxt+ Ri px — Rip 
— a 
sample error approximation error 


The first error, called sample error, is a probabilistic one since it depends on random 
samples, while the second error, called approximation error, is a deterministic one. 
Finding a good balance between both errors is sometimes called bias-variance 
problem, where the bias is related to the approximation error and the variance to 
the sampling error. 
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Concerning the approximation error, it turns out that for RKHS F = Hx on 
compact metric spaces XY which are dense in C() and continuous, P-integrable, 
so-called Nemitski losses, this error becomes zero; see Corollary 5.29 in Steinwart 
and Christmann [93]. In particular, this is true for RKHS with the Gaussian kernel 
and the loss functions considered in Sect. 3. For relations between the approximation 
error, interpolation spaces, and K-functionals, see the book of Cucker and Zhou [29] 
and the references therein. 

Concerning the sample error, there is a huge amount of results, and this paper 
can only cover some basic directions. For a survey on recent developments in 
the statistical analysis of classification methods, see Boucheron et al. [14]. Based 
on Hoeffding’s inequality, the first of such relations goes back to Vapnik and 
Chervonenkis [107]. See also the books of Tsypkin [104], Vapnik [105], Anthony 
and Bartlett [3], and Vidyasagar [109]. To get an impression how such estimates 
look like, two of them from Proposition 6.18 and 6.22 of the book of Steinwart and 
Christmann [93] are presented in the following. If F is finite and L(x, y, f(x)) < 
B, then it holds for all m > 1 that 


I + 2InQ\F 
(ze (4% x YY": Rip (fz) — Rt pp = B za rern <eVr>0. 
P, V m7 


If the function class F is infinite, in particular not countable, one needs some bounds 
on the “complexity” of F. The most classical of such a “complexity” measure 
is the VC dimension (see Vapnik and Chervonenkis [107]) applied in connection 
with the 0/1 loss function. Another possibility is the use of covering numbers or its 
counterpart entropy numbers going back to Kolmogorov and Tikhomirov [57]. The 
é-covering number of a metric set T with metric d is the size of the smallest e-net 
of T, 1.e., 


N(T, d,&) := it >1:4s),...,8, € T such that T C |_) Ba(s;.2) F 


i=1 


where By(s,€) is the closed ball with center s and radius e. For the estimation of 
covering numbers see Edmunds and Triebel [35] and Pinkus [79]. Then, for compact 
F C Leo(X), one has basically to replace |F| in the above relation by its covering 
number: 


p”™ (2 € (Xx VY)": Rip (fz) — Ri pg 


. a + 2In(2N(F, || - loos €)) 


+ iL] <e* 
m 
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for all t > O and for all ¢ > 0, where ones assumes in addition that || flo < M, 
Ff €¥F and that L is locally Lipschitz continuous with constant | L| jy; here. 

Next let us turn to the SVM setting, where an additional term comes along with 
the loss function, namely, one is interested in minimizers of 


ink (Ruz(P) + Allf leg)» 2 > 0 


with a regularization term A|| f Ee that penalizes functions f with large RKHS 
norms. The techniques developed for ERM analysis can be extended to the SVM 
setting. 

First let us mention that under some assumptions on the loss function, which are 
fulfilled for the setting in Sect. 3, a unique minimizer fz, exists and has the form 


m 


faa = >a; K(x, +). 


i=l 


This was established in the representer theorem by Kimeldorf and Wahba [56] for 
special continuous loss functions and generalized, e.g., in Schdlkopf et al. [86]. 
There also exist a representer-like theorems for the minimizer fp, of the infinite- 
sample setting 


ink (Rue (P) + Alf lie 


(see Steinwart [92], de Vito [111], and Dinuzzo et al. [33]). One can show for the 
infinite-sample setting that the error 


A(A) := ‘ 


inf {Rvp(f)+Allf liz} — Ripe 
CHK 


tends to zero as A goes to zero and that limy.9 Rz.p(fpa) = Rip 4ig: Let us 
come to the essential question how close Rp, (fz,,) is to R7 p. Recall that RT p = 


Ry PUx for the above mentioned RKHS. An ERM analysis like estimation has, for 
example, the form 


Pp” (z e(X x Vy": Ri p(fza) +All fall — Ripa, 


> AA) + AT? |Li-ae + YD 


2t + 2In(2N (Br, || + loo. A? 
i tT + 21n(2N(Brx, |I + lloo &)) + elton] Se", 
m 


for t > 0, where one assumes that the continuous kernel fulfills ||K]lo. < 1, 
L(x, y,0) < 1 and By, is the closed unit ball in H (see Theorem 6.25 in Steinwart 
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and Christmann [93] and also Cucker and Smale [28] and Bousquet and Elisseeff 
[15]). For a certain decay of the covering number In(2N(By,, || - |loo, €)) in € and 
a RKHS for which the approximation error becomes zero, one can then conclude 
that for zero sequences (Am)m>1 With an additional suitable decay property related 
to the decay of the covering number, the relation Ry p(fz.1,,) > Ri p holds true 
in probability. 

The above relations can be further specified for classification and regression tasks 
with special loss functions. With respect to classification, one can find, for example, 
upper bounds for the risk in terms of the margin or the number of support vectors. 
For the 0/1 loss function, the reader may consult, e.g., the book of Cristianini and 
Shawe-Taylor [27]. For the hinge loss function and the soft margin SVM with C = 
1/(2Am), it holds, for example, that 


Ts 
USI 5 OAM fealPug + Ruz fea) 

(see Proposition 8.27 in Steinwart and Christmann [93]). For a suitable zero 
sequence (A»)m>1 and a RKHS with zero approximation error, the following 
relation is satisfied: 


Ii F(Z) > OS pe 


lim P”™ (z e(¥ xy)": 
m 


— °) =1, e«>0. 

m—>Co 

Finally, let us address the setting, where the risk function defining the learning 
task is hard to handle numerically. One example is the risk function associated with 
the 0/1 loss function. This function is neither continuous nor convex. One remedy 
is to replace such unpleasant loss functions L by a convex surrogate Lsy, where one 
has to ensure that the minimizer fz in (43) for the surrogate loss fulfills Rr p (fz) * 
Rj. p. For the hinge function as surrogate of the 0/1 loss function, Zhang [119], has 
proved that 


Rion .P(S) ~ RiP = R1,.P (f) = Rip 


for all measurable functions f. Thus, if Rz,.p (fz) — Ri, p is small, this follows 
for the original risk function, too. For a systematical treatment of surrogate loss 
functions, the reader may consult Chapter 3 in the book of Steinwart and Christmann 
[93] which was partially inspired by the work of Barlett et al. [6]. 


5 Numerical Methods 


This section concentrates on the support vector machines in Sect.3. Numerical 
methods for the other models were always sketched when they were introduced. 
Support vector machines require finally the minimization of a quadratic func- 
tional subject to linear constraints (QP). These minimization problems involve a 
symmetric, fully populated kernel matrix having the size m of the training set. 
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Hence, this matrix has in general m(m + 1)/2 distinct nonzero coefficients one has 
to work with. Therefore, one has to distinguish between small- to moderate-sized 
problems, where such a matrix can be stored into the RAM of the computer, and 
large-sized problems, say, with more than a million training data. 

For quadratic programming with small to moderate data sizes, there exist various 
meanwhile standard algorithms. They are implemented in commercial software 
packages like CPLEX or MOSEK (see also the MATLAB optimization toolbox or in 
freeware packages like MINOS and LOQO). Among them, the primal-dual interior 
point algorithms belong to the most reliable and accurate techniques. The main idea 
of interior point methods is to solve the primal and dual problems simultaneously 
by enforcing the Kuhn-Tucker conditions to iteratively find a feasible solution. The 
duality gap, i.e., the difference between the minimum of the primal problem and the 
maximum of the dual problem, is used to determine the quality of the current set of 
variables and to check whether the stopping criteria are fulfilled. For QP algorithms 
including recent algorithms for solving large QPs, the reader may consult the new 
book of Zdenek [118]. 

The problem of learning large data sets was mainly addressed based on so-called 
“working set” methods. The idea is the following: if one knew in advance which 
constraints were active, it would be possible to cancel all of the inactive constraints 
which simplifies the problem. 

The simplest method in this direction is known as chunking. It starts with an 
arbitrary subset (“chunk” = working set) of the data and trains the SVM using 
an optimizer on this subset. The algorithm then keeps the support vectors and 
deletes the others. Next, the M points (VM algorithm parameter) from the remaining 
part of the data, where the “current SVM” makes the largest errors, are added to 
these support vectors to form a new chunk. This procedure is iterated. In general, 
the working set grows until in the last iteration the machine is trained on the 
set of support vectors representing the active constraints. Chunking techniques in 
SVMs were already used by Vapnik [106] and were improved and generalized in 
various papers. 

Currently, more advanced “working set” methods, namely, decomposition algo- 
rithms, are one of the major tools to train SVMs. These methods select in each 
iteration a small fixed size subset of variables as working set, and a QP problem 
is solved with respect to this set (see, e.g., Osuna et al. [77]). A special type of 
decomposition methods is the sequential minimal optimization (SMO) which uses 
only working sets of two variables. This method was introduced by Platt [80] for 
classification; see Flake and Lawrence [42] for regression. The main advantage of 
these extreme small working sets is that the partial QP problems can be solved 
analytically. For the soft margin SVM in the dual form from Sect. 3 (with a variable 
exchange a +> Ya) 


1 
5 «Ka —(y,a) subjectto (ln,@) =0,0< ya <C. 


the SMO algorithm looks as follows: 
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SMO-type decomposition methods 
1. Fix a as initial feasible solution and set k := 1. 


2. Ifa) solves the dual problem up to a desired precision, stop. 
Otherwise, select a working set B := {i, 7} C {1,...,m}. Define N := 
{1,...,m}\B 


“) 


and ap and a) as sub-vectors of a) corresponding to B and N, resp. 


3. Solve the following subproblem with fixed a) for ap: 


(k)y\t\ {Kee Ken aB aB 
3 + (a, (ay )" ) 6a =) (<5) _ (v5 yy) (<5) 


Kii K; 
= 5 (0% @;) ( i) C ‘) — (aj a;) (Kava\” - 7) + constant —> 
Kij Kjj 


subject to aj + a;= =I, ee 0< Vidi, VjAj S C. 


Set wer to be the minimizer. 
4. Set a +) =a), k +> k + 1 and goto Step 2. 


The gaa iis solution in Step 3 is given as follows: for simplicity, set B := 
-li, 500) and (cj cj)" := Kpnaw — yz. Substituting vw; = 6 — a; from the 
first constraint into the objective function, one gets 


1 
5 i (Ki —2Kj; + Kj;) + 0;(BKi; — BK; —¢; +¢;) + constant > min. 


Or; 


If K is positive definite, it holds that K;; —2K;; + K;; > 0, and the above function 
has a unique finite global minimizer a;,,. One has to take care about the second 
constraint. This constraint requires that a; € [L,U], where L and U are defined by 


(max(0,6—C),min(C,8)) ify = ly; =1, 
(max(0, 6),min(C,B+C)) ify =1,y; =—-1, 
(max(—C, 6B — C), min(0, 8)) if y; = -l,y; = 1, 
(max(—C, B), min(0, 8 + C)) if y; = -l,y; =—-1. 


(L,U):= 


Hence the minimizer in Step 3 is given by (a, B — a*), where 


Qig if Qig € [L,U], 
a :=)L ifaig <L, 
U if dig > U. 
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It remains to determine the selection of the working set. (The determination of the 
stopping criteria is beyond the scope of this paper.) Indeed, current decomposition 
methods vary mainly according to different working set selections. The SVM'#" 
algorithm of Joachims [52], was originally based on a rule for the selection 
the working set of Zoutendijk [120]. Moreover, this algorithm uses a shrinking 
technique to speed up the computational time. Shrinking is based on the idea 
that if a variable a") remains equal to zero or C for many iteration steps, then 
it will probably not change anymore. The variable can be removed from the 
optimization problem such that a more efficient overall optimization is obtained. 
Another shrinking implementation is used in the software package LIBSVM of 
Chang and Lin [22]. A modification of Joachims’ algorithm for regression called 
“SVMTorch” was given by Collobert and Bengio [25]. An often addressed working 
set selection due to Keerthi et al. [55] is the so-called maximal violating pair 
strategy. A more general way of choosing the two-element working set, namely, 
by choosing a constant factor violating pair, was given, including a convergence 
proof, by Chen et al. [24]. For convergence results, see also the paper of Lin [65]. 
The maximal violating pair strategy relies on first-order (i.e., gradient) information 
of the objective function. Now for QP, second-order information directly relates 
to the decrease of the objective function. The paper of Fan et al. [38] proposes a 
promising working set selection based on second-order information. 

For an overview of SVM solvers for large data sets, the reader may also consult 
the books of Huang et al. [50] and Bottou et al. [13] and the paper of Mangasagian 
and Musicant [69] with the references therein. An extensive list of SVM software 
including logistic loss functions and least squares loss functions can be found on the 
webpages www.kernel-machines.org and www.support-vector-machines.org. 


6 Conclusion 


The invention of SVMs in the 1990s led to an explosion of applications and 
theoretical results. This paper can only give a very basic introduction into 
the meanwhile classical techniques in this field. It is restricted to supervised 
learning although SVMs have also a large impact on semi- and unsupervised 
learning. 

Some new developments are sketched as multitask learning where, in contrast to 
single-task learning, only limited work was involved until now and novel techniques 
taken from convex analysis come into the play. 

An issue that is not addressed in this paper is the robustness of SVMs. There is 
some ongoing research on connections between stability, learning, and prediction of 
ERM methods (see, e.g., the papers of Elisseeff et al. [36] and Mukherjee et al. [74]). 

Another field that has recently attained attention is the use of kernels as diffusion 
kernels on graphs (see Kondor and J. Lafferty [58] and also the book of Shawe- 
Taylor and Cristianini [88]). 
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Abstract 


The use of total variation as a regularization term in imaging problems was 
motivated by its ability to recover the image discontinuities. This is on the basis 
of his numerous applications to denoising, optical flow, stereo imaging and 3D 
surface reconstruction, segmentation, or interpolation, to mention some of them. 
On one hand, we review here the main theoretical arguments that have been given 
to support this idea. On the other hand, we review the main numerical approaches 
to solve different models where total variation appears. We describe both the 
main iterative schemes and the global optimization methods based on the use 
of max-flow algorithms. Then we review the use of anisotropic total variation 
models to solve different geometric problems and its use in finding a convex 
formulation of some non-convex total variation problems. Finally we study the 
total variation formulation of image restoration. 


1 Introduction 


The total variation model in image processing was introduced in the context 
of image restoration [55] and image segmentation, related to the study of the 
Mumford—Shah segmentation functional [34]. Being more related to our purposes 
here, let us consider the case of image denoising and restoration. 

We assume that the degradation of the image occurs during image acquisition 
and can be modeled by a linear and translation invariant blur and additive noise: 


f=hxutn, (1) 


where u : R? — R denotes the ideal undistorted image, h : R? > R is a blurring 
kernel, f is the observed image which is represented as a function f : R* > R, and 
n is an additive white noise with zero mean and standard deviation o. In practice, 
the noise can be considered as Gaussian. 

A particular and important case contained in the above formulation is the 
denoising problem which corresponds to the case where h = 4, so that Eq. (1) 
is written as 


f =utn, (2) 

where n is an additive Gaussian white noise of zero mean and variance o°. 
The problem of recovering u from f is ill-posed. Several methods have been 
proposed to recover u. Most of them can be classified as regularization methods 
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which may take into account statistical properties (Wiener filters), information 
theoretic properties [35], a priori geometric models [55], or the functional analytic 
behavior of the image given in terms of its wavelet coefficients (see [48] and 
references therein). 

The typical strategy to solve this ill-conditioning is regularization [56]. In the 
linear case the solution of (1) is estimated by minimizing a functional 


Jy(u) =|| Hu— f |lz + || Qu [h3. (3) 


which yields the estimate 
uy = (H'H + yQ'Q)"H' f, (4) 


where Hu = h x u, and Q is a regularization operator. Observe that to obtain 
uy, we have to solve a system of linear equations. The role of Q is, on one 
hand, to move the small eigenvalues of H away from zero while leaving the 
large eigenvalues unchanged, and, on the other hand, to incorporate the a priori 


(smoothness) knowledge that we have on wu. 


1/2 pl/2 


If we treat u and n as random vectors and we select y = | and OQ = R, “ R, 
with R,; and R,, the image and noise covariance matrices, then (4) corresponds to the 
Wiener filter that minimizes the mean square error between the original and restored 
images. 

One of the first regularization methods consisted in choosing between all possible 
solutions of (1) the one which minimized the Sobolev (semi) norm of u 


i |Du|? dx, (5) 
R2 


which corresponds to the case Qu = Du. In the Fourier domain the solution of 


(3) given by (4) isa = ee ee fd . From the above formula we see that high 
h|-+4ym- 


frequencies of f (hence, the noise) are attenuated by the smoothness constraint. 
This formulation was an important step, but the results were not satisfactory, 
mainly due to the inability of the previous functional to resolve discontinuities 
(edges) and oscillatory textured patterns. The smoothness required by the finiteness 
of the Dirichlet integral (5) constraint is too restrictive. Indeed, functions in 
W!?(R?) (i.e., functions u € L?(IR*) such that Du € L*(R?)) cannot have discon- 
tinuities along rectifiable curves. These observations motivated the introduction of 
Total Variation in image restoration problems by L. Rudin, S. Osher, and E. Fatemi 
in their work [55]. The a priori hypothesis is that functions of bounded variation (the 
BV model) [9] are a reasonable functional model for many problems in image pro- 
cessing, in particular, for restoration problems [55]. Typically, functions of bounded 
variation have discontinuities along rectifiable curves, being continuous in some 
sense (in the measure theoretic sense) away from discontinuities [9]. The disconti- 
nuities could be identified with edges. The ability of total variation regularization 
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to recover edges is one of the main features which advocates for the use of this 
model, but its ability to describe textures is less clear, even if some textures can be 
recovered, up to a certain scale of oscillation. An interesting experimental discussion 
of the adequacy of the BV -model to describe real images can be found in [41]. 

In order to work with images, we assume that they are defined in a bounded 
domain Q C R? which we assume to be the interval [0, N[?. As in most of the 
works, in order to simplify this problem, we shall assume that the functions / and 
u are periodic of period N in each direction. That amounts to neglecting some 
boundary effects. Therefore, we shall assume that /, u are functions defined in Q 
and, to fix ideas, we assume that h,u € L?(Q). Our problem is to recover as much 
as possible of u, from our knowledge of the blurring kernel h, the statistics of the 
noise n, and the observed image /. 

On the basis of the BV model, Rudin—Osher—Fatemi [55] proposed to solve the 
following constrained minimization problem: 


Minimize / |Du| 
Q 
(6) 
subject to / |h * u(x) — f(x)? dx < 07 |Q|. 
Q 


Notice that the image acquisition model (1) is only incorporated through a global 
constraint. Assuming that / * 1 = 1 (energy preservation), the additional constraint 
that [, hxudx = f, f(x) is automatically satisfied by its minima [28]. In practice, 
the above problem is solved via the following unconstrained minimization problem: 


1 
Minimize | |Du| + =f |nxu— f dx (7) 
Q 2A Q 


where the parameter A is positive. Recall that we may interpret 4 as a penalization 
parameter which controls the trade-off between the goodness of fit of the constraint 
and the smoothness term given by the Total Variation. In this formulation, a 
methodology is required for a correct choice of A. The connections between (6) 
and (7) were studied by A. Chambolle and P.L. Lions in [28] where they proved that 
both problems are equivalent for some positive value of the Lagrange multiplier A. 
In the denoising case, the unconstrained variational formulation (7) with h = 6 is 


1 
Minimize [|u| + >> [ w= Fax (8) 
Q 21 Jo 


and it has been the object of much theoretical and numerical research (see [10, 56] 
for a survey). Even if this model represented a theoretical and practical progress in 
the denoising problem due to the introduction of BV functions as image models, 
the experimental analysis readily showed its main drawbacks. Between them, let 
us mention the staircasing effect (when denoising a smooth ramp plus noise, the 
staircase is an admissible result), the pixelization of the image at smooth regions, 
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and the loss of fine textured regions, to mention some of them. This can be 
summarized with the simple observation that the residuals f —u, where u represents 
the solution of (8), do not look like noise. This has motivated the development of 
nonlocal filters [17] for denoising, the use of a stochastic optimization technique to 
estimate u [47], or the consideration of the image acquisition model as a set of local 
constraints [3, 38] to be discussed below. 

Let us finally mention that, following the analysis of Y. Meyer in [48], the 
solution u of (8) permits to obtain a decomposition of the data f as a sum of two 
components u+v where u contains the geometric sketch of f while v is supposed to 
contain its noise and textured parts.As Meyer observed, the L* norm of the residual 
v := f —u in (8) is not the right one to obtain a decomposition of f in terms of 
geometry plus texture and he proposed to measure the size of the textured part v in 
terms of a dual BV norm showing that some models of texture have indeed a small 
dual BV norm. 

In spite of its limitations, the total variation model has become one of the basic 
image models and has been adapted to many tasks: optical flow, stereo imaging 
and 3D surface reconstruction, segmentation, interpolation, or the study of u + v 
models to mention a few cases. On the other hand, when compared to other robust 
regularization terms, it combines simplicity and geometric character and makes it 
possible a rigorous analysis. The theoretical analysis of the behavior of solutions of 
(8) has been the object of several works [6, 13, 14,20,48,50] and will be summarized 
in Sects. 3 and 4. 

Recall that one of the main reasons to introduce the Total Variation as a 
regularization term in imaging problems was its ability to recover discontinuities 
in the solution. This intuition has been confirmed by the experimental evidence and 
has been the motivation for the study of the local regularity properties of (8) in 
[20, 23]. After recalling in Sect.2 some basic notions and results in the theory of 
bounded variation functions, we prove in section “The Discontinuities of Solutions 
of the TV Denoising Problem” that the set of jumps (in the BV sense) of the solution 
of (8) is contained in the set of jumps of the datum f [20]. In other words, model (8) 
does not create any new discontinuity besides the existing ones. As a refinement of 
the above statement, the local Hélder regularity of the solutions of (8) is studied in 
section “Hélder Regularity Results.” This has to be combined with results describing 
which discontinuities are preserved. No general statement in this sense exists, but 
many examples are described in the papers [5, 10, 13, 14]. The preservation of a jump 
discontinuity depends on the curvature of the level line at the given point, the size 
of the jump, and the regularization parameter A. This is illustrated in the example 
given in Sect. 4. The examples support the idea that total variation is not perfect but 
may be a reasonable regularization term in order to restore discontinuities. 

Being considered as a basic model, the numerical analysis of the total variation 
model has been the object of intensive research. Many numerical approaches have 
been proposed in order to give fast, efficient methods which are also versatile to 
cover the whole range of applications. In Sect.5 we review some basic iterative 
methods introduced to solve the Euler-Lagrange equations of (8). In particular, we 
review in section “Chambolle’s Algorithm” the dual approach introduced by A. 
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Chambolle in [25]. In section “Primal-Dual Approaches” we review the primal- 
dual scheme of Zhu and Chan [57]. Both of them are between the most popular 
schemes by now. In Sect.6 we discuss global optimization methods based on 
graph-cut techniques adapted to solve a quantized version of (8). Those methods 
have also become very popular due to its efficiency and versatility in applications 
and are an active area of research, as it can be seen in the references. Then, in 
section “Global Solutions of Geometric Problems” we review the applications of 
anisotropic TV problems to find the global solution of geometric problems. Similar 
anisotropic TV formulations appear as convexifications of nonlinear energies 
for disparity computation in stereo imaging, or related problems [30, 52], and 
they are reviewed in section “A Convex Formulation of Continuous Multilabel 
Problems.” 

In Sect. 8 we review the application of Total Variation in image restoration (6), 
describing the approach where the image acquisition model is introduced as a set of 
local constraints [3, 38, 54]. 

We could not close this chapter without reviewing in Sect.9 a recent algorithm 
introduced by C. Louchet and L. Moisan [47] which uses a Bayesian approach 
leading to an estimate of u as the expected value of the posterior distribution of u 
given the data f. This estimate requires to compute an integral in a high dimensional 
space and the authors use a Monte-Carlo method with Markov Chain (MCMC) [47]. 
In this context, the minimization of the discrete version of (8) corresponds to a 
Maximum a Posterior (MAP) estimate of uw. 


2 Notation and Preliminaries on BV Functions 
Definition and Basic Properties 


Let Q be an open subset of R”. Let u € Li,-(&). Recall that the distributional 
gradient of u is defined by 


[o-pu= -| u(x)div a(x) dx VC%°(Q,R%), 
Q Q 


where C°°(Q; IR") denotes the vector fields with values in RY which are infinitely 
differentiable and have compact support in Q. The total variation of u in Q is 
defined by 


V(u, Q) := sup} [ udiva dx :0 € C%(Q;RY), |o(x)|<1V¥xEeQh, ©) 
Q 


where for a vector v = (vj,..., vv) € R% we set |v|? := >. v?. Following the 


usual notation, we will denote V(u, 2) by | Du|(Q) or by f, |Dul. 
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Definition 1. Let u ¢ L'(Q). We say that u is a function of bounded variation in 
Q if V(u, 2) < co. The vector space of functions of bounded variation in Q will be 
denoted by BV({2). 


Using Riesz representation Theorem [9], the above definition can be rephrased 
by saying that wu is a function of bounded variation in Q if the gradient Du in the 
sense of distributions is a (vector valued) Radon measure with finite total variation 
V(u, Q). 

Recall that BV(Q2) is a Banach space when endowed with the norm |u|] := 
Jglul dx + |Du\(Q). Recall also that the map u + |Du|(Q) is Lj,.(@)-lower 
semicontinuous, as a sup (9) of continuous linear forms [9]. 


Sets of Finite Perimeter: The Co-area Formula 


Definition 2. A measurable set FE C Q is said to be of finite perimeter in Q if 
Xe € BV(Q). The perimeter of FE in Q is defined as P(E, Q) := |Dye|(Q). If 
Q = RY’, we denote the perimeter of E in R” by P(E). 


The following inequality holds for any two sets A, B C Q: 
P(AU B,Q) + P(AN B,Q) < P(A, Q) + P(B,&). (10) 


Theorem 1. Let u € BV(Q). Then for a.e. t € R the set {u > t} is of finite 
perimeter in QQ and one has 


[ |Du| = [- P(fu > t},Q) dt. 


In other words, the total variation of u amounts to the sum of the perimeters of its 
upper level sets. 


An analogous formula with the lower level sets is also true. For a proof we refer 
to [9]. 


The Structure of the Derivative of a BV Function 


Let us denote by £% and 1", respectively, the N -dimensional Lebesgue measure 

and the (N — 1)-dimensional Hausdorff measure in R% (see [9] for precise 

definitions). 
Let u € [Ly,.(Q)]” (m = 1). We say that w has an approximate limit at x € Q if 


there exists € € IR” such that 


1 
lim ———— u(y) — Edy = 0. (11) 
pi0 | B(x, p)| fea 0) — Slay 
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The set of points where this does not hold is called the approximate discontinuity 
set of u, and is denoted by S,. Using Lebesgue’s differentiation theorem, one can 
show that the approximate limit € exists at C’’-a.e. x € Q, and is equal to u(x), in 
particular, |S,,| = 0. If x € Q \ S,, the vector € is uniquely determined by (11) and 
we denote it by u(x). 

We say that u is approximately continuous at x if x ¢ S, and u(x) = u(x), that 
is, if x is a Lebesgue point of u (with respect to the Lebesgue measure). 

Let u € [Lj,,(Q)]” and x € Q \ S,; we say that u is approximately differentiable 
at x if there exists anim x N matrix L such that 


lu(y) = (2) = LO =I 4, 


lim ———_ = 0. (12) 
po | B(x, p)| B(x,p) p 


In that case, the matrix L is uniquely determined by (12) and is called the 
approximate differential of u at x. 

For u € BV(QQ), the gradient Du is a N-dimensional Radon measure that 
decomposes into its absolutely continuous and singular parts Du = D“u + D*u. 
Then D“u = Vu dx where Vu is the Radon—Nikodym derivative of the measure 
Du with respect to the Lebesgue measure in R" . The function u is approximately 
differentiable £' -a.e. in Q and the approximate differential coincides with Vu(x) 
LN -a.e. The singular part D°u can be also split into two parts: the jump part D/u 
and the Cantor part D°u. 

We say that x € Q is an approximate jump point of u if there exist ut (x) # 
u- (x) € Rand |v,,(x)| = 1 such that 


1 


Lt a a u(y) —u* (x)| dy = 0 
pyo |By (x, Vu(x))| Byt (x,vu(x)) 


1 


im —W——_ lu(y) —u (x)| dy = 0, 
po |Bo (x, Vu(x))| By (x.vulx)) 


where B(x; Vu(x)) = {y € Bx, p) : (y — x, vu(x)) > Of and By (x, vu(x)) = 
{y € B(x, p) : (y —x, vu(x)) < 0}. We denote by J, the set of approximate jump 
points of u. If u € BV(Q), the set S,, is countably #%—! rectifiable, J, is a Borel 
subset of S,,, and HY—!(S, \ J,) = 0 [9]. In particular, we have that 1% —!-a.e. 
x € Q is either a point of approximate continuity of u or a jump point with two 
limits in the above sense. Finally, we have 


Diu= D*ul_ 5, = (ut =a en. J, 


u 


and D°u = D*ul_ (Q\S,) 


For a comprehensive treatment of functions of bounded variation we refer to [9]. 
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3 The Regularity of Solutions of the TV Denoising Problem 
The Discontinuities of Solutions of the TV Denoising Problem 


Given a function f € L*(Q) and A > 0 we consider the minimum problem 


; 1 2 
vanin,, [1a + 5 [u-P dx. (13) 


Notice that problem (13) always admits a unique solution u,, since the energy 
functional is strictly convex. 

As we mentioned in Sect. 1, one of the main reasons to introduce the Total 
Variation as a regularization term in imaging problems was its ability to recover 
the discontinuities in the solution. This section together with section “Hdélder 
Regularity Results” and Sect. 4 is devoted to analyze this assertion. In this section 
we prove that the set of jumps of uw, (in the BV sense) is contained in the set 
of jumps of f, whenever f has bounded variation. Thus, model (13) does not 
create any new discontinuity besides the existing ones. Section “Hélder Regularity 
Results” is devoted to review a local H6lder regularity result of [23]: the local 
Holder regularity of the data is inherited by the solution. This has to be combined 
with results describing which discontinuities are preserved. In Sect.4 we give an 
example of explicit solution of (13) which shows that the preservation of a jump 
discontinuity depends on the curvature of the level line at the given point, the size 
of the jump, and the regularization parameter A. Other examples are given in the 
papers [2,5, 10, 13, 14]. The examples support the idea that total variation may be a 
reasonable regularization term in order to restore discontinuities. 

Let us recall the following observation, which is proved in [5, 18,24]. 


Proposition 1. Let u, be the (unique) solution of (13). Then, for any t € R, {u, > 
t} (respectively, {u, > t}) is the minimal (resp., maximal) solution of the minimal 
surface problem 


1 
min P(E, 2) + ; fe fem dx (14) 


(whose solution is defined in the class of finite-perimeter sets, hence up to a 
Lebesgue-negligible set). In particular, for all t € R but a countable set, {u, = t} 
has zero measure and the solution of (14) is unique up to a negligible set. 


A proof that {u, > t} and {u, > t} both solve (14) is found in [24, Prop. 2.2]. 
A complete proof of this proposition, which we do not give here, follows from the 
co-area formula, which shows that, up to a renormalization, for any u € BV(Q) N 
L*(Q), 
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[\ou+ x fw Pas -[(e (u> 1,9) +5 ft es) dt, 


and from the following comparison result for solutions of (14) which is proved in 
(5, Lemma 4]: 


Lemma 1. Let f,g € L'(Q) and E and F be respectively minimizers of 
min P(E, &2) -| f(x) dx and min P(F,Q) -{ g(x)dx. 
E E F F 


Then, if f < ga.e., |E \ F| = 0 (in other words, E C F up to a negligible set). 


Proof. Observe that we have 


PE.) | f(x)dx = PEENF.9)- f F(x) dx 


P(F.Q)— | g(x) dx < PEEUF.Q~ | g(a)dx, 


Adding both inequalities and using that for two sets of finite perimeter we have (10) 
P(E) F,Q)+ P(E U F,Q) < P(E, Q) + P(F, Q), we obtain that 


y (g(x) — f(x) dx <0. 
E\F 


Since g(x) — f(x) > 0 ae., this implies that E \ F is a null set. Oo 


The proof of this last lemma is easily generalized to other situations (Dirichlet 
boundary conditions, anisotropic and/or nonlocal perimeters; see [5] and also [2] for 
a similar general statement). Eventually, we mention that the result of Proposition 1 
remains true if the term (u(x) — f(x))*/(2A) in (13) is replaced with a term of the 
form W(x, u(x)), with Y of class C! and strictly convex in the second variable, and 
replacing (t — f(x))/A with 0, (x, ft) in (14). 

From Proposition | and the regularity theory for surfaces of prescribed curvature 
(see for instance [8]), we obtain the following regularity result (see also [2]). 


Corollary 1. Let f € L?(Q), with p > N. Then, for all t € R the super-level 
set E, := {uy > t} (respectively, {u, > t}) has boundary of class C', for all 
a < (p—N)/p, out of a closed singular set X& of Hausdorff dimension at most 
N —8. Moreover, if p = 0, the boundary of E, is of class W?4 out of ©, for all 
q < ©, and is of class C'! if N = 2. 
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We now show that the jump set of u, is always contained in the jump set of f. 
Before stating this result let us recall two simple lemmas. 


Lemma 2. Let U be an open set in RN andv € W~?(U), p = 1. We have that 


w( ues (y) = Trace (A(Vv(y)) D?v(y)) ae.inU, 


V1 |Vv|2 


N 
h A = 1 (Ce a) : RY. 
where AG) = ene CU OHM yar * S 


The proof follows simply by taking g ¢ C>°(U), integrating by parts in U, and 
regularizing v with a smoothing kernel. 


Lemma 3. Let U be an open set in RN and v € W*'!(U). Assume that u has a 
minimum at yo € U and 


lim 
p>0+ Bo(yo) 


| |u(y) — u(yo) — VuCyo)-(» — yo) — $(D?v(90)(y — yo), ¥ — Yo) | 
Bp (v0) e 


dy =0. 
(15) 

Then D*v(yo) = 0. 

If A is a symmetric matrix and we write A > 0 (respectively, A < 0) we mean 
that A is positive (resp., negative) semidefinite. 

The result follows by proving that H’—!-a.e. for € in S—! (the unit sphere in 
IR”) we have (D?v(yo)é, €) > 0. 

Recall that if vy € W*!(U), then (15) holds a.e. on U [58, Theorem 3.4.2]. 


Theorem 2. Let f € BV(Q)N L®(Q). Then, for all A > 0, 
Jy = Jy (16) 


up to a set of zero H® —!-measure). 
p 


Before giving the proof let us explain its main idea which is quite simple. Notice 
that, by (14), formally the Euler-Lagrange equation satisfied by dF; is 


1 
es) = 0 on 0F;, 
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where Kz, is the sum of the principal curvatures at the points of dE,. Thus if x € 
Jy, \ Jf, then we may find two values fy < fz such that x € 0E;, NOE; \ Jr. Notice 
that E,, C E;, and the boundaries of both sets have a contact at x. Of the two, the 
smallest level set is the highest and has smaller mean curvature. This contradicts its 
contact at x. 


Proof. Let us first recall some consequences of Corollary 1. Let E, := {uy > ft}, 
t € R, and let &, be its singular set given by Corollary 1. Since f € L°(Q), around 
each point x € dE; \ D;,t € R, dE; is locally the graph of a function in W*? for 
all p € [1, 00) (hence C!* for any a € (0, 1)). Moreover, if Vi := Useo &,, then 
HNN) = 0. 

Let us prove that H"—!(J,,, \ J) = 0. Observe that we may write [9] 


jg |). dE, dE, 


ty teQt <tr 


Thus it suffices to prove that for all t), t2 € Q, t) < t2, we have 
HN! (0E, NOE, \ (NU Jz)) = 0. (17) 


Let us denote by B’, the ball of radius R > 0 in R‘~! centered at 0. Let Cr := Bh x 
(—R, R). Let us fix th, €Q, tr < th. Given x € 0E,, N OE;, \ N, by Corollary 1, 
we know that there is some R > 0 such that, after a change of coordinates that aligns 
the x-axis with the normal to dE;, M dE;, at x, we may write the set JE;, 1 Cr as 
the graph of a function v; € W?? (Bb), Vp & [1,oo), x = (0,v;(0)) € Cr € Q, 
Vvi(0) = 0,7 € {1,2}. Without loss of generality, we assume that v; > 0 in Bp, 
and that F;, is the supergraph of v;,i = 1,2. From t; < f and Lemma |, it follows 
E,, © E,,, which gives in turn v2 > v; in Bb. 

Notice that, since 0E;, is of finite H—! measure, we may cover JE;, NOE,, \N 
by a countable set of such cylinders. Thus, it suffices to prove that 


HN! (GE, NOE, Cr) \ (NU Jp)) = 0. (18) 
holds for any such cylinder Cz as constructed in the last paragraph. 


Let us denote the points x € Cz as x = (y,z) € Bh x (—R, R). Then (18) will 
follow if we prove that 


HN —|(Mp) = 0, (19) 
where 
Mr := {y € Br: vi(y) = v2(y)} \ Ly € Br: (vi (y)) € Je}. 


Recall that, by Theorem 3.108 in [9], HN —!-ae. in y € Bk, the function 
fy. -) € BV(R, R)) and the jumps of f(y, -) are the points z such that 
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(y.z) € Jy. Recall that v; is a local minimizer of 


1 v(y) 
ming) =f Vi+TPay-> ff w= so.aazay, 
R R 


By taking a positive smooth test function y(y) of compact support in B,, and 
computing lim,_+o4 1 (Ei\(v + ev) — Ej(v)) = 0, we deduce that 


: Vvi(y) 1 N-1 , 

div + — (t; — f(y, vi(y) + 0) < 0, H—'-a.e. in By. 
Vi+lWiQyP 4 ‘ 

(20) 


In a similar way, we have 


VV; 1 
div ily) 75 (t; — f(y, vi(y) — 0) = 0, H!-ae. in Bp. 


V¥1+1Vvi(y)/? os 


Finally we observe that since v1, v2 € WP (B) for any p € [1,0o) and v2 > vy 
in By, by Lemma 3 we have that D?(v; — v2)(y) < 0H !-ae. on {y € By: 
vi(y) = v2(y)}- 

Thus, if #~'(Mzp) > 0, then there is a point y € Ma such that Vv,(¥) = 
Vv2(¥), D?(v1 — v2)(¥) < 0, f(¥, -) is continuous at v1(~) = v2(¥), and both 
Eqs. (20) and (21) hold at y. As a consequence, using Lemma 2 and subtracting the 
two equations, we obtain 


0 > trace(A(Vv1 (¥)) D7v1 (F)) — trace(A(Vv2(¥))D7v2(¥)) = ° qi + > 0, 


This contradiction proves (19). oO 


Holder Regularity Results 


Let us review the local regularity result proved in [23]: if the datum / is locally 
Hdlder continuous with exponent 6 € [0, 1] in some region Q’ C Q, then a local 
minimizer u of (13) is also locally Hélder continuous in Q’ with the same exponent. 

Recall that a function vu € BV(Q) is a local minimizer of (13) if for any v € 
BV(Q2) such that u — v has support in a compact subset K C Q, we have 


1 1 
[ious [wco-sepPax = fons; [ wo-seoPax 2) 
K 2JK K 2J/K 
It follows that u satisfies the equation [18] 
—divztu = f (23) 


with z € L©(Q,R) with ||zIloo < 1, and z- Du = |Dul [10]. 
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As in section “The Discontinuities of Solutions of the TV Denoising Problem” 
[20], the analysis of the regularity of the local minimizers of u is based on the 
following observation: for any ¢ € R, the level sets {u > t} (resp., {u > t}) are 
solutions (the minimal and maximal, indeed) of the prescribed curvature problem 
(14) which is defined in the class of finite-perimeter sets and hence up to a Lebesgue- 
negligible set. The local regularity of uw can be described in terms of the distance 
of any two of its level sets. This is the main idea in [20] which can be refined to 
obtain the Holder regularity of solutions of (14). As we argued in section “The 
Discontinuities of Solutions of the TV Denoising Problem,” outside the jump 
discontinuities of # (modulo an #1‘ ~!-null set), any two level sets at different 
heights cannot touch and hence the function u is continuous there. To be able to 
assert a Hélder type regularity property for u one needs to prove a local estimate 
of the distance of the boundaries of two level sets. This can be done here under the 
assumption of local Holder regularity for f [23]. 


Theorem 3. Let N < 7 and let u be a solution of (23). Assume that f is in C°? 
locally in some open set A © Q, for some B € [0,1]. Then u is also C°* locally 
in A. 


The Lipschitz case corresponds to 8 = 1. 

One can also state a global regularity result for solutions of the Neumann 
problem when Q C R% is a convex domain. Let f : Q — R be a uniformly 
continuous function, with modulus of continuity wy : [0,++00o) — [0, +00), that 
is, | f(x) — f(Y)| < @¢ (|x — yl) for all x,y € Q. We consider the solution u 
of (23) with homogeneous Neumann boundary condition, that is, such that (22) for 
any compact set K C Q and any v € BV(Q) such that v = u out of K. This solution 
is unique, as can be shown adapting the proof of [18, Cor. C.2.] (see also [10] for the 
required adaptations to deal with the boundary condition), which deals with the case 
Q=R"., 

Then, the following result holds true [23]: 


Theorem 4. Assume N < 7. Then, the function u is uniformly continuous in Q, 
with modulus @, < wf. 


Again, it is quite likely here that the assumption N < 7 is not necessary for this 
result. 


4 Some Explicit Solutions 


Recall that a convex body in R“ is a compact convex subset of R”. We say that a 
convex body is nontrivial if it has nonempty interior. 

We want to exhibit the explicit solution of (13) when f = yc and C isa 
nontrivial convex body in R. This will show that the preservation of a jump 
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discontinuity depends on the curvature of dC at the given point, the size of the 
jump, and the regularization parameter J. 
Let uy ,c be the unique solution of the problem: 


1 
min / ul + 55 f (u— yc) dx. (24) 
u€BV(RY) J RN 2r RYN 


The following result was proved in [5]. 


Proposition 2. We have that0 < wc < 1,uc = Oin RY \ C, and uy.c¢ is 
concave in {uj.c > O}. 


The proof of 0 < uy. < 1 follows from a weak version of the maximum 
principle [5]. Thanks to the convexity of C, by comparison with the characteristic 
function of hyperplanes, one can show that u;,c = 0 out of C [5]. To prove that 
uj.c is concave in {u,,c > 0} one considers first the case where C is of class cil 
and A > 0 is small enough. Then one proves that u,,c is concave by approximating 
uj,c by the solution u, of 


u — Adiv we _. inC 
Ve + |Vul? 
Vu 


VE + [Vue 


as € — 0+, using Korevaar’s concavity Theorem [46]. Then one considers the case 
where C is of class C'! and we take any A > 0. In this case, the concavity of uy.¢ in 
{u,.c > 0} is derived after proving Theorems 5 and 6 below. The final step proceeds 
by approximating a general convex body C by convex bodies of class C!! [4]. 

Moreover, since uy,c = 0 out of C, the upper level set {uy .¢ > s} C C for any 
s € (0, 1]. Then, as in Proposition 1, one can prove that for any s € (0, 1] the level 
set {uy c > s} is a solution of 


(25) 


vo =0 in OC, 


(P), min P(E) ~ IE. (26) 


for the value of « = A~!'(1 — s). When taking A € (0,+00) ands € (0, 1] we 
are able to cover the whole range of jz € [0, 00) [5]. By Lemma | we know that if 
p< p’ and C,,, Cy are minimizers of (P),, (P),’, respectively, then C,, C Cy’. 
This implies that the solution of (P), is unique for any value jz € (0,00) up toa 
countable exceptional set. Thus the sets C,, can be identified with level sets of uj .¢ 
for some A > 0 and, therefore, we obtain its uniqueness from the concavity of u).c. 
One can prove [4, 5,21]: 
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Theorem 5. There is a value 4* > 0 such that 


ifu<p*, Cy =9, 
if u > uw, C, is unique (and convex), 


if 4 = *, there are two solutions % and C,,* , 


where C,,« is the unique Cheeger set of C. Moreover for any i < ||xc||x we have 


be = 1lls.clles and C,, := {uj,c > 1— pA} for any > p*, where 


IXelle = max} [ uycdx: ue BV(R*), / |Du| < i ; 
RN RN 


The set C,,* coincides with the level set {uy,c = |\ua,c |loo} and is of class C'". 


We call a Cheeger set in a nonempty open bounded subset Q of R% any set 
G C Q which minimizes 
P(F) 


‘= mi ‘ 27 
Co aa (27) 


The Theorem contains the assertion that there is a unique Cheeger set in any 
nonempty convex body of R” and u* = Cg. This result was proved in [21] for 
uniformly convex bodies of class C*, and in [4] in the general case. Notice that the 
solution of (24) gives a practical algorithm to compute the Cheeger set of C. 


Theorem 6. Let C be a nontrivial convex body in RN. Let 
no —inffu:xeC, if xEeCc 
0 if x ER \C. 


Then uy.c(x) = (1 + AHc(x))* xc. 


If N = 2 and yw > yp’, the set C,, coincides with the union of all balls of 
radius 1/j contained in C [6]. Thus its boundary outside dC is made by arcs of 


circle which are tangent to dC. In particular, if C is a square, then the Cheeger set 
P(C,,*) ai 
IC*| OR 
see that the corners of C are rounded and the discontinuity disappears as soon as 
2 > 0 (see the left part of Fig. 1). This is a general fact at points of dC where its 


mean curvature is infinite. 


corresponds to the arcs of circle with radius R > 0 such that . We can 


Remark I. By adapting the proof of Proposition 4 in [5] one can prove the following 
result. If Q is a bounded subset of R% with Lipschitz continuous boundary, and 
u € BV(Q) N L?(Q) is the solution of the variational problem 
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Fig. 1 Left: The denoising of a square. Right: The Cheeger set of a cube 


1 
min tf |Du| + a fw was+ f jane (28) 
ueBV(2)NL2(2) Jo 21 Jo aa 


then 0 < u < 1 and for any s € (0, 1] the upper level set {u > 5} is a solution of 


min P(F)—A'(-s)|F|. (29) 


If A > Ois big enough, indeed greater than 1 /||7o||, then the level set {u = ||u||o} 
is the maximal Cheeger set of (2. In particular, the maximal Cheeger set can be 
computed by solving (28), and for that we can use the algorithm in [25] described in 
section “Chambolle’s Algorithm.” In the right side of Fig. 1 we display the Cheeger 
set of a cube. 


Other explicit solutions corresponding to the union of convex sets can be found in 
(2, 13]. In particular, Allard [2] describes the solution corresponding to the union 
of two disks in the plane and also the case of two squares with parallel sides 
touching by a vertex. Some explicit solutions for functions whose level sets are a 
finite number of convex sets in R? can be found in [14]. 


5 Numerical Algorithms: Iterative Methods 


Notation 


Let us fix our main notations. We denote by X the Euclidean space R*. The 
Euclidean scalar product and the norm in X will be denoted by (-, -)y and ||- |lx, 
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respectively. Then the image u € X is the vector u = (uj, a j=v and the vector 


field € is the map € : {1,...,N} x {1,...,N} — R?. To define the discrete total 
variation, we define a discrete gradient operator. If u € X, the discrete gradient is a 
vectorin Y = X x X given by 


Vu := (Vxu, Vy), 


where 
Uj+1,j — Ui,j ifi<WN 
ViW)ij = of 30 
(a 0 iff =N, oY 
Uujj+1—Uuj; if j < N 
Vv i= ij+l i,j 31 
( yt) J 0 ifj=N (31) 
fori, 7 = 1,..., N. Notice that the gradient is discretized using forward differences 


and V+-+u could be a more explicit notation. For simplicity we have preferred to 
use Vu. Other choices of the gradient are possible; this one will be convenient for 
the developments below. 

The Euclidean scalar product in Y is defined in the standard way by 


@by= > 6L8,+2,8) 


1<i,j<N 


for every & = (€!, £2), E = (E', £2) € Y. The normof € = (£!, €2) € Y is, as usual, 
Elly = (&, Ey? We denote the Euclidean norm of a vector v € R? by |v|. Then 
the discrete total variation is 


Ja(u) = ||Vully = So (Vwi il (32) 


1<i,j<N 
We have 


Ja(u) = sup (§, Vu)y. (33) 
fey, |& jS1 VJ) 


By analogy with the continuous setting, we introduce a discrete divergence div 
as the dual operator of V, i.e., for every & € Y andu € X we have 


(—div &, u)x a (&, Vu)y. 
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One can easily check that div is given by 


1 —§Lyy fl <i<N 
(div €);,; => Ei, if — 


(34) 
7; -Gj1ifl<j<N 
2 : oo 
+ gi i . a 1 
—§P , if J =HN 
for every € = (&',&) EY. 
We have 
Ja(u) := max (u, div &), (35) 
fev 
where 
V=EeEY:[& /P -1<0, Vise {1,...,.N} 
Chambolle’s Algorithm 
Let us describe the dual formulation for solving the problem: 
in Sulu) + solu f13 36) 
min Ja(u a) Y 


where f € X. Using (35) we have 
in Ja(u) + lla — fl = min max (u,divé) + = lle FIR 
jer eee gg NE ae eee ne age oe 


= = max min (w, div€) + ale Fix: 
geV ue 


Solving explicitly the minimization in u, we have u = f — Adivé. Then 


A 
max min (uw, div €) + le filly = max (f. div §) — 5 IIdiv Il 


feV ue 
4 2 
= ——min | f ; 
2 eV x 


div§ — — 
ivé rT 


oi 
a 


xX 
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Thus if €* is the solution of 


2 
div € — f 


q (37) 


min 
£eV 


XxX 


then u = f — Adiv &* is the solution of (36). 
Notice that div &* is the projection of £ onto the convex set 


Ka := {divé :|& |< 1, Vij €(,..., NP. 


As in [25], the Karush-Kuhn—Tucker Theorem yields the existence of Lagrange 
multipliers @;,; = 0 for the constraints € € VY, such that we have for each (7, j) € 
{occoy NP 


V [div € _ yaaa aller _ a ij = 0, (38) 


with either @}; > 0 and |g,;| = 1, or aw; = 0 and |§,;| < 1. In the later case, we 
have V[div € — A7! f];,; = 0. In any case, we have 


at, = |V[divé -A" f],,;|- (39) 


Let v > 0, €° = 0, p => 0. We solve (38) using the following gradient descent 
(or fixed point) algorithm: 


gPtl — EP. + vV[divé? —A7" f];, ; — v|Vidiv €? — As fips leP (40) 


1] By? 
hence 


pti _ &? + vV[div €? —A7! f]j,; 
‘J T+ v|V[divé? —A-! f ij | 


(41) 


Observe that |&? | <1 foralli, 7 € {1,...,N}andevery p > 0. 


Theorem 7. In the discrete framework, assuming that v < 1, then div £? converges 


to the projection of £ onto the convex set Kq. If div &* is that projection, then 
u= f —Adiv&* is the solution of (36). 


In Fig.2 we display some results obtained using Chambolle’s algorithm with 
different set of parameters, namely, A = 5, 10. 

Today, the algorithms Nesterov [49], Beck and Teboulle [12], or the primal-dual 
approaches described in the next section provide more efficient ways to solve this 
dual problem. 
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Fig. 2 Denoising results obtained with Chambolle’s algorithm. (a) Top left: the original image. 
(b) Top right: the image with a Gaussian noise of standard deviation o = 10. (c) Bottom left: the 
result obtained with A = 5. (d) Bottom right: the result obtained with A = 10 


Primal-Dual Approaches 


The primal gradient descent formulation is based on the solution of (36). The dual 
gradient descent algorithm corresponds to (41). The primal-dual formulation is 
based on the formulation 


1 
minmax G(u, §) := (u,div§) + >> [lu — Ff \lx 


and performs a gradient descent in u and gradient ascent in &. 
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Given the intermediate solution (uw, &*) at iteration step k we update the dual 
variable by solving 


max G(u*, €). (42) 
geV 
Since the gradient ascent direction is VeG(uk ,€) = —Vu, we update & as 
AL _ k_ tka k 
get! — py (et vi), (43) 


where t;, denotes the dual stepsize and P, denotes the projection onto the convex 
set V. The projection Py, can be computed as in (41) or simply as 


§3, 


(Pv&)ij = max([€;,j|. 1)’ 


Now we update the primal variable u by a gradient descent step of 


min G(u, &*t!). (44) 
uEeX 


The gradient ascent direction is V,G(u, &*+!) and the update is 
ukt! = uk — & (AdivE*t! + uk — f), (45) 


where 6; denotes the primal stepsize. 

The primal-dual scheme was introduced in [57]. The convergence is empirically 
observed for a variety of suitable stepsize pairs (t,@) and is given in terms of 
the product t@. For instance, convergence is reported for increasing values 6; and 
TO, < 0.5, see [57]. This has been theoretically explained for a variant of the 
scheme proposed in [29], while a general convergence proof has been given in [15]. 

The primal gradient descent and the dual projected gradient descent method are 
special cases of the above algorithm. Indeed if one solves the problem (42) exactly 
(taking t, = 00 in (43)) the resulting algorithm is 


k+1 k _, vue k 
u =u" — & | —Adiv —— + u* —f }, (46) 
|Vur| 


with the implicit convention that we may take any element in the unit ball of R? 
when Vuk = 0. 

If we solve (44) exactly and still apply gradient ascent to (42), the resulting 
algorithm is 


gk+l = py, G + %V (div ef ty) , (47) 


which essentially corresponds to (41). 
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The primal-dual approach can be extended to the total variation deblurring 
problem 


. 1 
min Ja(u) + 55 ||Bu— flix (48) 


where f € X and B is a matrix representing the discretization of the blurring 
operator H. 
The primal-dual scheme is based on the formulation 


1 
a (u, div &) + 5 Bu - files (49) 


and the numerical scheme can be written as 


ai = Py (&* _ 1% Vu) 
yet! = uk _ fal (—div e*+! +4 AB! (Bui*! = f)) : 


(50) 


Since B is the matrix of a convolution operator, the second equation can be solved 
explicitly using the FFT. Convergence is empirically observed for a variety of 
suitable stepsize pairs (t,@) and is given in terms of the product 16, see [57] and 
a proof in [15]. See also [29] for an explicit rule for acceleration, with proofs of 
convergence. For a detailed study of different primal-dual methods we refer to [37]. 


6 Numerical Algorithms: Maximum-Flow Methods 


It has been noticed probably first in [51] that Maximal-flow/minimum-cut tech- 
niques could be used to solve discrete problems of the form (14), that is, to 
compute finite sets minimizing a discrete variant of the perimeter and an additional 
external field term. Combined with (a discrete equivalent of) Proposition 1, this 
leads to efficient techniques for solving (only) the denoising problem (8), including 
a method, due to D. Hochbaum, to compute an exact solution in polynomial time 
(up to machine precision). A slightly more general problem is considered in [27], 
where the authors describe in detail algorithms which solve the problem with an 
arbitrary precision. 


Discrete Perimeters and Discrete Total Variation 


We will call a discrete total variation any convex, nonnegative function J : RY > 
[0, +00] satisfying a discrete co-area formula: 


+00 
J(u) = i Ih) ds (51) 


foe) 


1478 V. Caselles et al. 


where x'"2*} © {0,1}” denotes the vector such that yes 
y= litu es: 


As an example we can consider the (anisotropic) discrete total variation 


= Oifu; < s and 


J@) = YS lwsiy—mislt+ Do luge asl (52) 
1<i<N 1<i<N 
1<j<N 1S j<N 
In this case u = (ui) pat can be written as a vector in RY” with M = N?. 


Then, (51) obviously holds since for any a,b € R, we have |a—-b| = 
fe |X tess) — Aforsy| As. 

Observe, on the other hand, that the discretization (36) does not enter this 
category (unfortunately). In fact, a discrete total variation will be always very 
anisotropic (or “crystalline”). 

We assume that J is not identically +-oo. Then, we can derive from (51) the 
following properties [27]: 


Proposition 3. Let J be a discrete total variation. Then: 


1. J is positively homogeneous: J(Au) = AJ(u) for any u € R™ andi = 0. 

2. J is invariant by addition of a constant: J(c1 + u) = J(u) for any u € R™ and 
c ER wherel=(1,...,De R™ is a constant vector, In particular, J(1) = 0. 

3. J is lower semicontinuous. 

4. p € dJ(u) & (WZER, p € OI (x=). 

5. J is submodular: for any u, u' € {0, 13, 


J(uvuw) + Juauw) < Ju) + JW’). (53) 

More generally, this will hold for any u,u’! € R™. 
Conversely, if J: {0,1}“ —> [0,+00] is a submodular function with J(0) = 
J(1) = 0, then the co-area formula (51) extends it to R™ into a convex function, 


hence a discrete total variation. 


If J is a discrete total variation, then the discrete counterpart of Proposition | 
holds: 


Proposition 4. Let J be a discrete total variation. Let f € R™ and letu € R™ be 
the (unique) solution of 


1 
min AJ(u) + =|lu— f |? (54) 
u€RM 2 


Then, for all s > 0, the characteristic functions of the super-level sets E; = {u > 
s} and E’ = {u > s} (which are different only if s € {uj,i = 1,...,M}) are 
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respectively the largest and smallest minimizer of 


M 
janin , AJ(0) + d 6:(s— fi). (55) 

The proof is quite clear, since the only properties which were used for showing 
Proposition | were (a) the co-area formula of Theorem | and (b) the submodularity 
of the perimeters (10). 

As a consequence, Problem (54) can be solved by successive minimizations 
of (55), which in turn can be done by computing a maximal flow through a 
graph, as will be explained in the next section. It seems that efficiently solving the 
successive minimizations has been first proposed in the seminal work of Eisner 
and Severance [36] in the context of augmenting-path maximum-flow algorithms. It 
was then developed, analyzed, and improved by Gallo, Grigoriadis, and Tarjan [39] 
for preflow-based algorithms. Successive improvements were also proposed by 
Hochbaum [42], specifically for the minimization of (54). We also refer to [26, 33] 
for variants, and to [45] for detailed discussions about this approach. 


Graph Representation of Energies for Binary MRF 


It was first observed by Picard and Ratliff [51] that binary Ising-like energies, that 
is, of the form 


> 016 — 8;1 — D> BiG, (56) 
i,j i 


a;,; = 0, Bi € R, 6 € {0, 1}, could be represented on a graph and minimized by 
standard optimization techniques, and more precisely using maximum-flow algo- 
rithms. Kolmogorov and Zabih [44] showed that the submodularity of the energy 
is a necessary condition, while, up to sums of ternary submodular interactions, it is 
also a sufficient condition in order to be representable on a graph. (But other energies 
are representable, and it does not seem to be known whether any submodular J can 
be represented on a graph, see [27, Appendix B] and the references therein.) 

In case J(u) has only pairwise interactions, as in (52), then Problem (55) has 
exactly the form (56), with a;,; = A if nodes i and j correspond to neighboring 
pixels, 0 else, and 6; is s — fj. 

Let us build a graph as follows: we consider V = {1,..., M} U {S} U {T} 
where the two special nodes S and T are respectively called the “source” and the 
“sink.” We consider then oriented edges (S,i) and (i, 7),i = 1,...,M, and (i, /), 
1 <i, 7 < M, and to each edge we associate a capacity defined as follows: 


c(S,i) = BF i=1,...,M, 
AL h tH lock, (57) 
M. 


c(i, J) = a,j 1<i,j < 
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Here B;* = max{0,6;} and 6; = max{0,—8;}, so that 8; = B* — B>. By 
convention, we consider there is no edge between two nodes if the capacity is zero. 
Let us denote by € the set of edges with nonzero capacity and by G = (V,€) the 
resulting oriented graph. 

We then define a “cut” in the graph as a partition of € into two sets S and 7, with 
S ¢SandT € 7. The cost of a cut is then defined as the total sum of the capacities 
of the edges that start on the source side of the cut and land on the sink side: 


C(S,7) = > c(u, Vv). 


(Hv)EE 
hES VET 
Then, if we let 6 € {0,1}” be the characteristic function of SM {1,..., M+, we 
have 
M M 
C(S,T) = D0 - 6B; + OB* + D> 0156 — 0;)* 
i=l ij=l 
M M M 
= 2 ot; (8 —6;)* + > 6B: + > AF 
ij=l i=l i=l 
If a;,; = aj; (but other situations are also interesting), this is nothing else than 


energy (56), up to a constant. 

Thus, the problem of finding a minimum of (56) [or (55)] can be reformulated as 
the problem of finding a minimal cut in the graph. Very efficient algorithms are 
available, based on a duality result of Ford and Fulkerson [1]. It states that the 
maximum flow on the graph constrained by the capacities of the edges is equal to 
the minimal cost of a cut. The problem reduces then to find the maximum flow in the 
graph. This is precisely defined as follows: starting from S, we “push” a quantity 
(x) along the oriented edges (u,v) € € of the graph, with the constraint that 
along each edge, 


O < Xyy < c(p,v) 


and that each “interior” node i must satisfy the flow conservation constraint 
) Xwi = 5 Xiu 
Le ue 


(while the source S' only sends flow to the network, and the sink T only receives). 
It is clear that the total flow f(x) = )0; xs; = >¢; xi,r which can be sent is 
bounded from above, and not hard to show that a bound is given by a minimal- 
cost cut (S, 7). The duality theorem of Ford and Fulkerson expresses the fact that 
this bound is actually reached by the maximal flow (x,y) (u,v)ee (which maximizes 
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J (x)), and the partition (S,7) is obtained by cutting along the saturated edges 
(u,v), where x,» = cy while x,,, = 0. 

We can find starting from S the first saturated edge along the graph, and cut there, 
or do the same starting from T and scanning the reverse graph: for 6; = s — fj, 
this will usually give the same solution except for a finite number of levels s, which 
correspond exactly to the levels {u; : i = 1,..., M} of the solution of (54) and are 
called the “breakpoints.” 

Several efficient algorithms are available to compute a maximum flow in 
polynomial time [1]. Although the time complexity of the algorithm in [16], of 
Boykov and Kolmogorov, is not polynomial, this algorithm seems to outperform 
others in terms in time computations, as it is particularly designed for the graphs 
with low connectivity which arise in image processing. 

The idea of a “parametric maximum-flow algorithm” [39] is to reuse the same 
graph (and the “residual graph” which remains after a run of a max-flow algorithm) 
to solve problems (55) for increasing values s € {50,51,...,5,}. This is easily 
shown to solve (54) up to an arbitrary precision (and in polynomial time, see [39]). 
It seems this idea was already present in a paper of Eisner and Severance [36]. 

However, it was shown in [42] by D. Hochbaum that in fact the exact solution 
to (54) can be computed, also in polynomial time. Let us now explain the basic idea 
of this approach; for details we refer to [27,42]. 

Let u = (u;)M , be the (unique) solution of (54). Proposition 4 tells us that as s 
varies, problem (55) has the same solution y*“="} as long as s does not cross any of 
the values {u; : i = 1,..., M}, which are precisely the breakpoints. 

Assume we have found, for two levels s; < 52, solutions 9! > 6? of (55) and 
assume also that these solutions differ. It means that there is a breakpoint u;, in 
between: there is at least one location i (and possibly other) with 5; < uj, < 52. 

Suppose for a while that the value u;, were the only breakpoint between s; and 
Sq (that is, at no other location i, we can have both s; < u;, < sz and uj, # uj, ). 

In this case, for s € [s1, 52], the optimal energy should be 


II 


F(s) 


M M 
Fi(s) = (10- Soars] + sy e 


i=l i=1 


if s < uj, and 


F(s) 


II 


Fr(s) 


II 


M M 
(x10 ->> 1) +s) 6 


i=1 i=1 


for s > uj. And the value uj, is the necessary (only) solution of the equation 
F (ig) = F2(uio)- 

Observe that in any case, as 6! > 6? and they are different, the slope of the affine 
function F;(s) is strictly above the slope of the affine function F2(s). Since also 
F(s1) < F(s;) (as @! is optimal for s,) and Fy(s2) < F,(s2), there is always a 
(unique) value s3 € [s), s2] for which F,(s3) = F2(s3). 
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The idea of the algorithm is now clear: we have to compute a new maximal flow 
(which, in fact, reuses the residual flows from the computations of 6! and 67) to 
solve (55) for the level s = 53. We find a solution 6°, of energy 


M M 
Fy(s3) = (110) evs] +93) 6 


i=l i=l 
Then, there are two cases: 


e Either F3(s3) = F\(s3) = F2(s3): in this case we have found a breakpoint, and 
there is no other in the interval [s;, s2]. Hence, the level sets {u > s} have been 
found for all values 5 € [s1, 52]: hes = 6! fors € [s,, 3] and 6 fors € [53, 52]. 

* Or F3(s3) < Fi (s3) = F2(s3). Then, in particular, it must be that the solution 67 
differs from both 6! and 6? (otherwise the energies would be the same). Hence 
we can start again to try solving the problem at the levels s4 and s5 which solve 
F\(s4) = F3(s4) and F3(s5) = F2(s5). Now, since there are only a finite number 
of possible sets @ solving (55) (bounded by M, as the solutions are nonincreasing 
with 5s), this situation can occur at most a finite number of times, bounded by M. 


In practice, this can be done in a very efficient way, using “residual graphs” to 
start the new maximal-flow algorithms, and to compute efficiently the new levels 
where to cut (there is no need, in fact, to compute the values AJ(0) + 0; 6; f; and 
>>; 9; for this). See [27,42] for details. 

For experimental results in the case of total variation denoising we refer to [26, 
27,33, 40]. 


7 Other Problems: Anisotropic Total Variation Models 
Global Solutions of Geometric Problems 


The theory of anisotropic perimeters developed in [7] permits to extend model (28) 
to general anisotropic perimeters, including as particular cases the geodesic active 
contour model with an inflating force [19,43], and a model for edge linking [22]. 
This permits to find the global minima of geometric problems that appear in image 
processing [22, 24, 27, 32]. 


The Anisotropic Total Variation and Perimeter Let us define the general notion 
of total variation with respect to an anisotropy. Following [7] we say that a function 
@ : 2x RN = (0, 00) is a metric integrand if ¢ is a Borel function satisfying the 
conditions: 


for a.e. x € Q, the map & € R™ > ¢(x, €) is convex, (58) 
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b(x,té) = |t|o(x,8) Wee Q, VEER, VreR, (59) 

and there exists a constant A > 0 such that 
0< (x8) <Allél| VxeQ, VEER. (60) 
We could be more precise and use the term symmetric metric integrand, but for 
simplicity we use the term metric integrand. Recall that the polar function ¢° : 


Q x RY > RU {+00} of ¢ is defined by 


o°(x, &*) = sup{(é*, &) : & E RY O(x, €) < 1}. (61) 


The function #°(x, -) is convex and lower semicontinuous. 
Let 


Kg(Q) := {6 € X—(Q): $°(x,0(x)) < 1fora.e. x € Q, [o-v%] = 0}. 


Definition 3. Let u € L!(Q). We define the ¢-total variation of u in Q as 
/ |Dulg := sup ff udiva dx: 0 €KP(Q),, (62) 
Q Q 


We set BVg(Q) := {u € L'(Q): te |Du|s < co} which is a Banach space when 
endowed with the norm |u|gyv,(9) °= fo |uldx + fo |Dulg. 


We say that E C RN has finite ¢-perimeter in Q if yz € BVg(Q). We set 
PoE.) := | Dx 
Q 


If Q = R”, we denote Pg(E) = Pil ER”). By assumption (60), if E C R% has 
finite perimeter in Q it has also finite ¢-perimeter in Q. 


A Variational Problem and Its Connection with Geometric Problems Let ¢ : 
Q x RY — R be a metric integrand in Q andh € L®(Q), h(x) > 0 ae., with 
te 76) dx < oo. Let us consider the problem 


min [ioue+ f blew lant +> f nu fPds, (63) 
Q a 2 Jo 


u€BVg (2) 


where v®% denotes the outer unit normal to dQ. To shorten the expressions inside 
the integrals we shall write h,u instead of h(x), u(x), with the only exception of 
(x, v@). The following result was proved in [22]. 
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Theorem 8. (i) Let f € L*(Q,hdx), i.e, fg f(x)? h(x) dx < 00. Then there is 
a unique solution of the problem (63). 

(ii) Ifu € BVg(Q) N L?(Q,h dx) be the solution of the variational problem (63) 
with f = 1. ThenO < u < 1 and the level sets E, := {x € Q: u(x) = 5}, 
s € (0, 1], are solutions of 


yun Po(F) — wIF la. (64) 


where |F\, = J; h(x) dx. As in the Euclidian case, the solution of (64) is 
unique for any s € (0, 1] up to a countable exceptional set. 
(iii) When X is big enough, the level set associated with the maximum of u, 


{u = ||Ulloo}, is the maximal (¢, h)-Cheeger set of Q, i.e., is a minimizer of 
the problem 
_ of Po (F) a ; ' 
inf \F| : F C Q of finite perimeter, |F |, > 0} . (65) 
h 


The computation of the maximal (¢, /)-Cheeger set [together with the solution 
of the family of problems (64)] can be computed by adapting Chambolle’s algorithm 
[25] described in section “Chambolle’s Algorithm.” 


Example 1. We illustrate this formalism with two examples: (a) the geodesic active 
contour model and (b) a model for edge linking. 


(a) The geodesic active contour model. Let I : Q — Rt be a given image in 
L©(Q), G be a Gaussian function, and 


1 


J1+|ViGe 1D? 


(where in G * IJ we have extended J to R™ by taking the value 0 outside Q). 
Observe that g € C(Q) and inf,-g g(x) > 0. The geodesic active contour 
model [19,43] with an inflating force corresponds to the case where @(x, £) = 
g(x)|€| and |Du|y = g(x)|Du| and h(x) = 1, x € Q. The purpose of this 
model is to locate the boundary of an object of the image at the points where 
the gradient is large. The presence of the inflating term helps to avoid minima 
collapsing into a point. The model was initially formulated [19, 43] in a level 
set framework In this case we may write P,(F’) instead of Py(/’), and we have 
P,(F) := foe § dH", where 0* F is the reduced boundary of F [9]. 

In this case the Cheeger sets are a particular instance of geodesic active 
contour with an inflating force whose constant is ~Z = C. An interesting 
feature of this formalism is that it permits to define local Cheeger sets as local 
(regional) maxima of the function u. They are Cheeger sets in a subdomain of 
Q. They can be identified with boundaries of the image and the above formalism 


g(x) = (66) 
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Fig. 3 Geodesic active contours and edge linking experiments. The first row shows the images 
I to be processed. The first three columns correspond to segmentation experiments, the last three 
are edge linking experiments. The second row shows the weights g used for each experiment 


(white is 1, black is 0), in the first two cases g = (.,/1+|V(G* /)|?)~!, for the third 
g= 0.37(/0.1 +|V(G * I)|?)~!, and for the linking experiments g = ds, the scaled distance 
function to the given edges. The third row shows the disjoint minimum g-Cheeger sets extracted 
from u (shown in the background); there are 1,7,2,1,1, and 1 sets, respectively. The last linking 
experiment illustrates the effect of introducing a barrier in the initial domain (black square) 


permits to compute several active contours at the same time (the same holds true 
for the edge linking model). 

(b) An edge linking model. Another interesting application of the above formalism 
is to edge linking. Given a set ! C Q (which may be curves if Q C R? or 
pieces of surface if 2 C R*), we define dp(x) = dist(x, I’) and the anisotropy 
o(x,&) = dr(x)|&|. In that case, we experimentally see that the Cheeger set 
determined by this anisotropy links the set of curves (or surfaces) T’. If T is a 
set of edges computed with an edge detector we obtain a set or curves (NV = 2) 
or surfaces (NV = 3) linking them. 


Notice that, for a given choice of ¢, we actually find many local ¢-Cheeger sets, 
disjoint from the global minimum, that appear as local minima of the Cheeger ratio 
on the tree of connected components of upper level sets of u. The computation of 
those sets is partially justified by Proposition 6.11 in [22]. These are the sets which 
we show on the following experiments. 

Let us mention the formulation of active contour models without edges proposed 
by Chan—Vese in [31] can also be related to the general formulation (64). 

In Fig. 3, we display some local ¢-Cheeger sets of 2D images for the choices of 
metric ¢ corresponding to geodesic active contour models with an inflating force 
(the first three columns) and to edge linking problems (the last three columns). 
The first row displays the original images, and the second row displays the metric 
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g = (f/14+|/V(G*J)|2)~! or g = ds. The last row displays the resulting 
segmentation or set of linked edges, respectively. Let us remark here a limitation 
of this approach that can be observed in the last subfigure. Even if this linking 
is produced, the presence of a bottleneck (bottom right subfigure) makes the ds- 
Cheeger set to be a set with large volume. This limitation can be circumvented by 
adding barriers in the domain Q: we can enforce hard restrictions on the result by 
removing from the domain some points that we do not want to be enclosed by the 
output set of curves. 


A Convex Formulation of Continuous Multilabel Problems 


Let us consider the variational problem 


min fiea+ [ W(x, u(x)) dx, (67) 
u€BV(Q),0<u<M Jo Q 


where W : Q x R — R? is a potential which is Borel measurable in x and 
continuous in u, but not necessarily convex. Thus the functional is nonlinear and 
non-convex. The functional can be relaxed to a convex one by considering the 
subgraph of uv as an unknown. 

Our purpose is to write the nonlinearities in (67) in a “convex way” by intro- 
ducing a new auxiliary variable [52]. This will permit to use standard optimization 
algorithms. The treatment here will be heuristic. 

Without loss of generality, let us assume that M = 1. Let d(x, 5) = H(u(x)—s), 
where H = Yj0,400) 18 the Heaviside function and s € R. Notice that the set of 
points where u(x) > s (the subgraph of w) is identified as @(x,s) = 1. That is, 
(x, s) is an embedding function for the subgraphs of u. This permits to consider 
the problem as a binary set problem. The graphs of u is a “cut” in @. 

Let 


A:= {¢ € BV(Q x [0, 1]) : d(x, 5) € {0, 1}, V(x, 5) € Q x 0, I}. 


Using the definition of anisotropic total variation [7] we may write the energy in 
(67) in terms of @ as 


1 
min ff (|Dx.¢| + W(x, s)|0s6(x, s)|]) dx dt+ 
a Jo 


pEeA 


i (W(x, 0)16(x.0) — 1] + Wx, DIb(x. DI) ax. (68) 


where the boundary conditions @(x,0) = 1, @(x, 1) = 0 are taken in a variational 
sense. 
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Although the energy (68) is convex in @ the problem is non-convex since the 
minimization is carried on A which is a non-convex set. The proposal in [52] is to 
relax the variational problem by allowing ¢ to take values in [0, 1]. This leads to the 
following class of admissible functionals: 


A := {6 € BV(Qx(0, 1]) : o(x,s) € [0, 1], V(x, s) € 2x0, 1], 6; < 0}. (69) 


The associated variational problem is written as 


1 
min ff (D.01 + W0.s))8.b(e.s)) dx drt 
geA JQ/S0 


[ (W(x, 0)|@(x,0) — 1] + Wx, Die. D) dx. (70) 


This problem is now convex and can be solved using the dual or primal-dual 
numerical schemes explained in sections “Chambolle’s Algorithm” and “Primal- 
Dual Approaches.” Formally, the level sets of a solution of (70) give solutions of 
(67). This can be proved using the developments in [7, 22]. 

In [30] the authors address the problem of convex formulation of multilabel 
problems with finitely many values including (67) and the case of non-convex 
neighborhood potentials like the Potts model or the truncated total variation. The 
general framework permits to consider the relaxation in BV(Q) of functionals of 
the form 


F(u) =f fleu(x), Yule) dx (71) 


where u € W!!(Q) and f : Q x Rx RY — [0,oo[ be a Borel function such 
that f(x, z,€) is a convex function of & for any (x,z) € Q x R% satisfying some 
coercivity assumption in &. Let f* denote the Legendre—Fenchel conjugate of f 
with respect to &. If 


K := {¢ = (¢". ¢°): 2x R— R’: ¢ is smooth and 
F* (x, 8, 9 (x, 5)) = $" (x, 5)}. 


then the lower semicontinuous relaxation of F is 


F(u) = sup | [6 Daten: 
eeKJQJR 


Based on this formula one can use a dual or a primal-dual numerical scheme to 
minimize F (u) if one knows how to compute the projection onto the convex set K. 
We refer to [30,52] for details. 
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8 Other Problems: Image Restoration 


To approach the problem of image restoration from a numerical point of view we 
shall assume that the image formation model incorporates the sampling process in a 
regular grid 


fii =(hewij +71; (i,j) €{1,..., NY, (72) 


where u : R? — R denotes the ideal undistorted image, h : R? — R is a blurring 
kernel, f is the observed sampled image which is represented as a function f : 
AA cacles NY — R, and n;,; is, as usual, a white Gaussian noise with zero mean and 
standard deviation o. 

Let us denote by Qy the interval [0, NV [?. As we said in the introduction, in order 
to simplify this problem, we assume that /, u are functions defined in Qy and are 
periodic of period N in each direction. To fix ideas, we assume that h, u € L7(Q N)s 
so that / * uw is a continuous function in Qy and the samples (h * u);,;, (i,j) € 
{1,...,.N}, have sense. 

Let us define the discrete functional 


@= Yo JP+iVwijP. B=. 


1<i,j<N 
For any function w € L?(Qy), its Fourier coefficients are 


- (ixt+jy 
Wigs i wo ye O™ a a for (l,j) €Z. 
. as 


Our plan is to compute a band limited approximation to the solution of the 
restoration problem for (72). For that we define 


1 1 1 
B:= L’?(Qy) 2a i tedin}—-~+—,...,x=¢}. 
Jue (Qy) : uw is supported in a 7 34} 


We notice that B is a finite-dimensional vector space of dimension N* which can be 
identified with X. Both J(u) = /, ay |Du| and hy (u) are norms on the quotient space 
/R; hence they are equivalent. With a slight abuse of notation we shall indistinctly 
writeu € Borue X. : 
We shall assume that the convolution kernel h € L?(Qy) is such that h is 
supported in {-3 + y batts 5} and h(o, 0)=1. 
In the discrete framework, the ROF model for restoration is 


Minimize cx J? (u) (73) 
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N 
subjectto S° |(h* wij — ijl’ < oN’. (74) 


ij=l 


Notice again that the image acquisition model (1) is only incorporated through 
a global constraint. In practice, the above problem is solved via the following 
unconstrained formulation: 


a 


1 N 
5 | ye dy Ween - AP - © (75) 


ij=l 


min max J 4 (u) + 
uEX a0 


where a > 0 is a Lagrange multiplier. The appropriate value of a can be computed 
using Uzawa’s algorithm [3] so that the constraint (74) is satisfied. Recall that if 
we interpret a! as a penalization parameter which controls the importance of the 
regularization term, and we set this parameter to be small, then homogeneous zones 
are well denoised while highly textured regions will lose a great part of its structure. 
On the contrary, if a! is set to be small, texture will be kept but noise will remain 
in homogeneous regions. On the other hand, as the authors of [3] observed, if we 
use the constrained formulation (73)-(74) or, equivalently (75), then the Lagrange 
multiplier does not produce satisfactory results since we do not keep textures and 
denoise flat regions simultaneously, and they proposed to incorporate the image 
acquisition model as a set of local constraints. 
Following [3], we propose to replace the constraint (74) by 


Gx(h*xu—f)ij <0", VG, J) S{lcceg NY, (76) 


where G is a discrete convolution kernel such that G;,; > 0 for all (i,j) € 
{1,..., N}’. The effective support of G must permit the statistical estimation of the 
variance of the noise with (76) [3]. Then we shall minimize the functional J ‘3 (u) 
on X submitted to the family of constraints (76) (plus eventually the constraint 
eet * Wij = a Ji,;). Thus, we propose to solve the optimization 
problem: 


-. 7B 
min Ji (u) 
(77) 
subject to Gx(heu- fy, <o° V(i, j). 


This problem is well posed, i.e., there exists a solution and is unique if 8 > 0 and 
inf.cr G * (f —c)? > o?. Incase that B = 0 and inf.cg G * (f —c)* > o”, then 
h * u is unique. Moreover, it can be solved with a gradient descent approach and 
Uzawa’s method [3]. 

To guarantee that the assumptions of Uzawa’s method hold we shall use a 
gradient descent strategy. For that, let v € X and y > 0. At each step we have 
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to solve a problem like 


: ry B 
min y|u— vy + Jq (W) 


(78) 
subject to Ge(heu— fp, <0 VU, J). 


We solve (78) using the unconstrained formulation 


min max LY (u, {a}; v), 
ueX a>0 


where @ = (oi, =1 and 


N 
LY (u, {a}; v) = ylu—vik + JP) + D> on (G * (hau f)}j) — 07). 


ij=l 
Algorithm: TV-Based Restoration Algorithm with Local Constraints 


1. Set up = 0 or, better, uw = f.Setn = 0. 
2. Use Uzawa’s algorithm to solve the problem 


min max LY (u, {a}; u"), (79) 
uEX a>0 


that is: 
(a) Choose any set of values a, > 0,0, 7) € {1,..., NV}, and uv” = wu". Iterate 
from p = 0 until convergence of aw? the following steps: 
(b) With the values of aw? solve DP(y, uv”): 


min LY (u, {a}; uv") 


starting with the initial condition w’,. Let ul, 4 be the solution obtained. 
(c) Update a in the following way: 


oT! = max(a?, + o(G * (hx u",— f)?,-0°),0) VG. /). 


Let v"*! be the solution of (79). Stop when convergence of u”. 


We notice that, since y > 0, Uzawa’s algorithm converges if f € h * B. 
Moreover, if u° satisfies the constraints, then uv” tends to a solution wu of (77) as 
n—> oo [3]. 

Finally, to solve problem (79) in Step 2(b) of the Algorithm we use either the 
extension of Chambolle’s algorithm [25] to the restoration case if we use 6 = 0, or 
the quasi-Newton method as in [38] adapted to solve (79) when f > 0. For more 
details, we refer to [3,38] and references therein. 
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Some Restoration Experiments 


To simulate our data we use the modulation transfer function corresponding to SPOT 
5 HRG satellite with Hipermode sampling (see [53] for more details): 


A(m, no) = et Buln g—4taay m+ sine(2n,)sinc(2n2)sinc(m), (80) 


where 71, 2 € [—1/2, 1/2], sinc(y1) = sin(a1)/(772), a = 0.58, and B; = 0.14. 
Then we filter the reference image given in Fig. 4a with the filter (80) and we add 
some Gaussian white noise of zero mean and standard deviation o (in our case 
o = 1, which is a realistic assumption for the case of satellite images [53]) to 
obtain the image displayed in Fig. 4b. 

Figure 5a displays the restoration of the image in Fig.4b obtained using the 
Algorithm of last section with 8 = 0. We have used a Gaussian function G of 
radius 6. The mean value of the constraint is mean((G * (Ku— f))*) = 1.0933 and 
RMSE = 7.9862. Figure 5b displays the function a; ; obtained. 

Figure 6 displays some details of the results that are obtained using a single 
global constraint (74) and show its main drawbacks. Figure 6a corresponds to the 
result obtained with the Lagrange multiplier a = 10 (thus, the constraint (74) is 
satisfied). The result is not satisfactory because it is difficult to denoise smooth 
regions and keep the textures at the same time. Figure 6b shows that most textures 
are lost when using a small value of a (a = 2) and Fig. 6c shows that some noise 
is present if we use a larger value of a (a = 1,000). This result is to be compared 
with the same detail of Fig. 5a which is displayed in Fig. 6d. 


Fig. 4 Reference image and a filtered and noised image. (a) Left: reference image. (b) Right: the 
data. This image has been generated applying the MTF given in (80) to the top image and adding 
a Gaussian white noise of zero mean and standard deviation o = | 
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Fig. 5 Restored image with local Lagrange multipliers. (a) Left: the restored image corresponding 
to the data given in Fig. 4b. The restoration has been obtained using the Algorithm of last section 
with a Gaussian function G of radius 6. (b) Right: the function a@;,; obtained 


Fig. 6 A detail of the restored images with global and local constraints. Top: (a), (b), and (c) 
display a detail of the results that are obtained using a single global constraint (74) and show its 
main drawbacks. Figure (a) corresponds to the result obtained with the value of a such that the 
constraint (74) is satisfied, in our case a = 10. Figure (b) shows that most textures are lost when 
using a small value of a (a = 2) and Figure (c) shows that some noise is present if we use a larger 
value of a (a = 1,000). Bottom: (d) displays the same detail of Fig. 5a which has been obtained 
using restoration with local constraints 
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The Image Model 


For the purpose of image restoration the process of image formation can be modeled 
in a first approximation by the formula [53] 


f = O{T(h *u) +n}, (81) 


where u represents the photonic flux, / is the point spread function of the optical- 
sensor joint apparatus, I] is a sampling operator, i.e., a Dirac comb supported by the 
centers of the matrix of digital sensors, n represents a random perturbation due to 
photonic or electronic noise, and Q is a uniform quantization operator mapping R 
to a discrete interval of values, typically [0, 255]. 


The Modulation Transfer Function for Satellite Images We describe here a 
simple model for the Modulation Transfer Function of a general satellite. More 
details can be found in [53] where specific examples of MTF for different 
acquisition systems are shown. The MTF used in our experiments (80) corresponds 
to a particular case of the general model described below [53]. 

Recall that the MTF, which we denote by h , is the Fourier transform of the point 
spread function of the system. Let (71,72) € [—1/2, 1/2] denote the coordinates 
in the frequency domain. There are different parts in the acquisition system that 
contribute to the global transfer function: the optical system, the sensor, and the blur 
effects due to motion. Since each subsystem is considered as linear and translation 
invariant, it is modeled by a convolution operator. The kernel k of the joint system 
is thus the convolution of the point spread functions of the separated systems. 


— Sensors: In CCD arrays every sensor has a sensitive region where all the photons 
that arrive are integrated. This region can be approximated by a unit square 
[—c/2,c/2]° where c is the distance between consecutive sensors. Its impulse 
response is then the convolution of two pulses, one in each spatial direction. 
The corresponding transfer function also includes the effect of the conductivity 
(diffusion of information) between neighboring sensors, which is modeled by an 
exponential decay factor; thus: 


hs(m, 2) = sinc(n1c)sine(noc)e 27711! e=27B2eln2| 


where sinc(n,) = sin(1)/(271) and Bi, B2 > 0. 

— Optical system: The optical system has essentially two effects on the image: it 
projects the objects from the object plane to the image plane and degrades it. 
The degradation of the image due to the optical system makes that a light point 
source loses definition and appears as a blurred (small) region. This effect can be 
explained by the wave nature of light and its diffraction theory. Discarding other 
degradation effects due to the imperfect optical systems like lens aberrations [1 1], 
the main source of degradation will be the diffraction of the light when passing 
through a finite aperture: those systems are called diffraction limited systems. 
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Assuming that the optical system is linear and translation invariant we know 
that it can be modeled by a convolution operator. Indeed, if the system is linear 
and translation invariant, it suffices to know the response of the system to a light 
point source located at the origin, which is modeled by a Dirac delta function 6, 
since any other light distribution could be approximated (in a weak topology) by 
superpositions of Dirac functions. The convolution kernel is, thus, the result of 
the system acting on 6. 

If we measure the light intensity and we use a circular aperture the MTF is 
considered as an isotropic low-pass filter 


ho(m, 12) = eat ae n/m +73 a> 0. 


— Motion: each sensor counts the number of photons that arrive to its sensitive 
region during a certain time of acquisition. During the sampling time the system 
moves a distance t and so does the sensor; this produces a motion blur effect in 
the motion direction (d,, d2): 


hu (m,n) = sine(((m, n2), (di, d2))t). 


Finally, the global MTF is the product of each of these intermediate transfer 
functions modeling the different aspects of the satellite: 


h(n, 2) = hshohy. 


Noise We shall describe the typical noise in case of a CCD array. Light is 
constituted by photons (quanta of light) and those photons are counted by the 
detector. Typically, the sensor registers light intensity by transforming the number 
of photons which arrive to it into an electric charge, counting the electrons which the 
photons take out of the atoms. This is a process of a quantum nature and therefore 
there are random fluctuations in the number of photons and photoelectrons on the 
photoactive surface of the detector. To this source of noise we have to add the 
thermal fluctuations of the circuits that acquire and process the signal from the 
detector’s photoactive surface. This random thermal noise is usually described by a 
zero-mean white Gaussian process. The photoelectric fluctuations are more complex 
to describe: for low light levels, photoelectric emission is governed by Bose— 
Einstein statistics, which can be approximated by a Poisson distribution whose 
standard deviation is equal to the square root of the mean; for high light levels, 
the number of photoelectrons emitted (which follows a Poisson distribution) can 
be approximated by a Gaussian distribution which, being the limit of a Poisson 
process, inherits the relation between its standard deviation and its mean [11]. Ina 
first approximation this noise is considered as spatially uncorrelated with a uniform 
power spectrum, thus a white noise. Finally, both sources of noise are assumed to 
be independent. 
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Taken together, both sources of noise are approximated by a Gaussian white 
noise, which is represented in the basic equation (81) by the noise term n. The 
average signal to noise ratio, called the SNR, can be estimated by the quotient 
between the signals average and the square root of the variance of the signal. 

The detailed description of the noise requires knowledge of the precise system of 
image acquisition. More details in the case of satellite images can be found in [53] 
and references therein. 


9 Final Remarks: A Different Total Variation-Based 
Approach to Denoising 


Let us briefly comment on the interesting work [47] which interprets the total 
variation model for image denoising in a Bayesian way leading to a different 
algorithm based on stochastic optimization which produces better results. 

We work again in the discrete setting and consider the image model 


fj =ujtny; GsAeU,....NP, (82) 


where n;,; is a white Gaussian noise with zero mean and standard deviation o. 

The solution of (36) can be viewed as a Maximum a Posteriori (MAP) estimate of 
the original image u. Let 6 > 0 and let pg be the prior probability density function 
defined by 


pplu) « e Bla) ue X, 


where we have omitted the normalization constant. The prior distribution models 
the gradient norms of each pixel as independent and identically distributed random 
variables following a Laplace distribution. Although the model does not exactly fit 
the reality since high gradient norms in real images are concentrated along curves 
and are not independent, it has been found to be convenient and efficient for many 
tasks in image processing and we follow it here. 

Since the probability density of f given u is the density forn = f —u, then 


full 


P(f|u) xe 29? 


Using Bayes rule, the posterior density of u given f is 


l = fully ( 
paul f) = PS palw) = 5e a) (83) 


full 
-( LON 4 Bg () 
é 2 


where Z = du is the normalization constant making the 


RN? 
mass of pg(u|f) to be 1. Then the maximization of the a posteriori density (83) is 
equivalent to the minimization problem (36) provided that Bo? = A. 


1496 V. Caselles et al. 


Fig. 7 (a) Left: the result obtained by computing E(u|f) and Bo? = 4 = 20, 0 = 10 (image 
courtesy of Cécile Louchet). (b) Right: the result obtained using Chambolle’s algorithm with A = 
20 


The estimation of u proposed in [47] consists in computing the expected value of 
u given f: 


1 I _( Wtaalle 8 ru) 
cul =4 f urstafrdu= 4 fuel ao a) 


This estimate requires to compute an integral in a high dimensional space. In [47], 
the authors propose to approximate this integral with a Markov Chain Monte-Carlo 
algorithm (MCMC). In Fig. 7a we display the result of denoising the image in 
Fig. 2b which has a noise of standard deviation o = 10 with the parameter 6 = 2. 
In Fig. 7b we display the denoising of the same image using Chambolle’s algorithm 
with A = 20. Notice that in both cases the parameter A is the same. 


10 Conclusion 


We have given in this chapter an overview of recent developments on the total 
variation model in imaging. Its strong influence comes from its ability to recover the 
image discontinuities and is the basis of numerous applications to denoising, optical 
flow, stereo imaging and 3D surface reconstruction, segmentation, or interpolation 
to mention some of them. We have reported the recent theoretical progress on the 
understanding of its main qualitative features. We have also reviewed the main 
numerical approaches to solve different models where total variation appears. We 
have described both the main iterative schemes and the global optimization methods 
based on the use of max-flow algorithms. Then, we reviewed the use of anisotropic 
total variation models to solve different geometric problems and its recent use in 
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finding a convex formulation of some nonconvex total variation problems. We have 
also studied the total variation formulation of image restoration and displayed some 
results. We have also reviewed a very recent point of view which interprets the 
total variation model for image denoising in a Bayesian way, leading to a different 
algorithm based on stochastic optimization which produces better results. 
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Abstract 


Since their introduction in a classic paper by Rudin, Osher, and Fatemi (Physica 
D 60:259-268, 1992), total variation minimizing models have become one of 
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the most popular and successful methodologies for image restoration. New 
developments continue to expand the capability of the basic method in various 
aspects. Many faster numerical algorithms and more sophisticated applications 
have been proposed. This chapter reviews some of these recent developments. 


1 Introduction 


Images acquired through an imaging system are inevitably degraded in various 
ways. The types of degradation include noise corruption, blurring, missing values 
in the pixel domain or transformed domains, intensity saturation, jittering, etc. Such 
degradations can have adverse effects on high-level image processing tasks such as 
object detection and recognition. Image restoration aims at recovering the original 
image from its degraded version(s) to facilitate subsequent processing tasks. Image 
data differ from many other kinds of data due to the presence of edges, which 
are important features in human perception. It is therefore essential to preserve 
and even reconstruct edges in the processing of images. Variational methods for 
image restoration have been extensively studied in the past couple of decades. A 
promise of these methods is that the geometric regularity of the resulting images 
is explicitly controlled by using well-established descriptors in geometry. For 
example, smoothness of object boundaries can be easily manipulated by controlling 
their length. There has also been much research in designing variational methods 
for preserving other important image features such as textures. 

Among the various restoration problems, denoising is perhaps the most funda- 
mental one. Indeed, all algorithms for solving ill-posed restoration problems have 
to have some denoising capabilities either explicitly or implicitly, for otherwise 
they cannot cope with any error (noise) introduced during image acquisition or 
numerical computations. Moreover, the noise removal problem boils down to the 
fundamental problem of modeling natural images which has great impacts on any 
image processing tasks. Therefore, research on image denoising has been very 
active. 


2 Background 


Total variation (TV)-based image restoration models are introduced by Rudin, 
Osher, and Fatemi (ROF) in their seminal work [51] on edge-preserving image 
denoising. It is one of the earliest and best-known examples of variational partial 
differential equation (PDE)-based edge-preserving denoising models. In this model, 
the geometric regularity of the resulting image is explicitly imposed by reducing the 
amount of oscillation while allowing for discontinuities (edges). The unconstrained 
version introduced in [1] reads: 


inf [ivu+n [ w-frax 
uEL?(Q) JQ Q 
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Here, Q is the image domain, f : Q — RF is the observed noisy image, u: Q > R 
is the denoised image, and jz > 0 is a parameter depending on the noise level. The 
first term is the total variation (TV) which is a measure of the amount of oscillation 
in the resulting image uw. Its minimization would reduce the amount of oscillation 
which presumably reduces noise. The second term is the L* distance between u and 
jf, which encourages the denoised image to inherit most features from the observed 
data. Thus, the model trades off the closeness to f by gaining the regularity of 
u. The noise is assumed to be additive and Gaussian with zero mean. If the noise 
variance level o” is known, then the parameter jx can be treated as the Lagrange 
multiplier, restraining the resulting image to be consistent with the known noise 
level, i.e. fg(u— f)’dx = |Q\o? [16]. 

The ROF model is simple and elegant for edge-preserving denoising. Since its 
introduction, this model has ignited a great deal of research in constructing more 
sophisticated variants which can give better reconstructed images, designing faster 
numerical algorithms for solving the optimization problem numerically, and finding 
new applications in various domains. In a previous book chapter [21] published 
in 2005, the authors surveyed some recent progresses in the research of total 
variation-based models. The present chapter aims at highlighting some exciting 
latest developments in numerical methods and applications of total variation-based 
methods since the last survey. 


3 Mathematical Modeling and Analysis 


In this section, the basic definition of total variation and some of its variants 
are presented. Then, some recent TV-based mathematical models in imaging are 
reviewed. 


Variants of Total Variation 


Basic Definition 
The use of TV as a regularizer has been shown to be very effective for processing 
images because of its ability to preserve edges. Being introduced for different 
reasons, several variants of TV can be found in the literature. Some variants can 
handle more sophisticated data such as vector-valued imagery and matrix-valued 
tensors; some are designed to improve restoration quality, and some are modified 
versions for the ease of numerical implementation. It is worthwhile to review the 
basic definition and its variants. 

In Rudin, Osher, and Fatemi’s work [51], the TV of an image f : Q > FR is 
defined as 


[ivrlax 
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where 2 C R? is a bounded open set. Since the image f may contain discon- 
tinuities, the gradient V f must be interpreted in a generalized sense. It is well 
known that elements of the Sobolev space W!!(Q) cannot have discontinuities 
[2]. Therefore, the TV cannot be defined through the completion of the space 
C! of continuously differentiable functions under the Sobolev norm. The V/f is 
thus interpreted as a distributional derivative, and its integral is interpreted as a 
distributional integral [40]. Under this framework, the minimization of TV naturally 
leads to a PDE with a distribution as a solution. 

Besides defining TV as a distributional integral, other perspectives can offer some 
unique advantages. A set theoretical way is to define TV as a Radon measure of the 
domain {2 [50]. This has an advantage of allowing Q to be a more general set. But 
a more practical and simple alternative is the “dual formulation.” It uses the usual 
trick in defining weak derivatives — integration by parts — together with the Fenchel 
transform, 


[uvri=s} f, fovea 


where f € L'(Q) and div is the divergence operator. Using this definition, one can 
bypass the discussion of distributions. It also plays an important role in many recent 
works in dual and primal-dual methods for solving TV minimization problems. The 
space BV can now be defined as 


g€C! (Q2,R’), |g(x)| < 1Vxe Q (1) 


sv ayi=} fetal | iwsi<ool 


Equipped with the norm || f ||gy = || fllz: + JQ |V JS], this space is complete and is 
a proper superset of W!!(Q) [32]. 


Multichannel TV 

Many practical images are acquired in a multichannel way, where each channel 
emphasizes a specific kind of signal. For example, color images are often acquired 
through the RGB color components, whereas microscopy images consist of mea- 
surements of different fluorescent labels. The signals in the different channels 
are often correlated (contain redundant information). Therefore, in many practical 
situations, regularization of multichannel images should not be done independently 
on each channel. 

There are several existing ways to generalize TV to vectorial data. A review of 
some generalizations can be found in [20]. Many generalizations are very intuitive. 
But only some of them have a natural dual formulation. Sapiro and Ringach [52] 
proposed to define 


fis =f YL ivsirtdx=f whledx 


i=l 
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where f = (f1(x), fo(x),.-., fiz (X)) is the vectorial data with M channels. Thus, it 
is the integral of the Frobenius norm | - | of the Jacobian V f. The dual formulation 
given in [10] is 


wp) [ sivas ge C; (2,R™™), |g(x)|p < 1Vx Ee Qh, 


where ( f, div g) = wer fi div gi. 


Matrix-Valued TV 

In applications such as diffusion tensor imaging (DTI), the measurements at each 
spatial location are represented by a diffusion tensor, which is a 3 x 3 symmetric 
positive semi-definite matrix. Recent efforts have been devoted to generalize the 
TV to matrix-valued images. Some natural generalizations can be obtained by 
identifying an M x N matrix with an MN vector, so that a vector-valued total 
variation can be applied. This was done by Tschumperlé and Deriche [57], who 
generalized the vectorial TV of [7]. The main challenge is to preserve the positive 
definiteness of the denoised solution. This will be elaborated in section “Diffusion 
Tensors Images.” 

Another interesting approach proposed by Setzer et al. [54] is the so-called 
operator-based regularization. Given a matrix-valued function f = (fj; (x)), define 
a matrix function A : = (a;;) where a;; = |V fi;|’. Let (A) be the matrix obtained 
by replacing each eigenvalue A of A with JX. Then the total variation is defined to 
be fo |®(A)| -dx, where |-|- is the Frobenius norm. While this formulation seems 
complicated, its first variation turns out to have a nice simple formula. However, 
when combined with the ROF model, the preservation of positive definiteness is an 
issue. 


Discrete TV 
The ROF model is cast as an infinite-dimensional optimization problem over the 
BV space. To solve the problem numerically, one must discretize the problem at 
some stage. The approach proposed by Rudin et al. in [51] is to “optimize then 
discretize.” The gradient flow equation is discretized with a standard finite difference 
scheme. This method works very well in the sense that the numerical solution 
converges to a steady state which is qualitatively consistent with the expected result 
of the (continuous) ROF model. However, to the best of the authors’ knowledge, 
a theoretical proof of convergence of the numerical solution to the exact solution 
of the gradient flow equation as the grid size tends to zero is not yet available. 
A standard convergence result of finite difference schemes for nonlinear PDE is 
based on the compactness of TV-bounded sets in L! [46]. However, proving TV 
boundedness in two or more dimensions is often difficult. 

An alternative approach is to “discretize then optimize.” In this case, only one has 
to solve a finite-dimensional optimization problem, whose numerical solution can 
in many cases be shown to converge. But the convergence of the exact solution 
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of the finite-dimensional problems to the exact solution of the original infinite- 
dimensional problem is often hard to obtain too. So, both approaches suffer from 
the theoretical convergence problem. But the latter method has a precise discrete 
objective to optimize. 

To discretize the ROF objective, the fitting term is often straightforward. But the 
discretization of the TV term has a strong effect on the numerical schemes. The 
most commonly used versions of discrete TV are 


m—l|n—-1 


iv] 4 Gags) Uae he (2) 


i=1 j=l 


m—ln-1 


Ifllry = >) 0 (fitus — fui] + [fusti— fal) Ax, (3) 


i=1 j=l 


where f = (fj,;) is the discrete image and Ax is the grid size. They are sometimes 
referred as the isotropic and anisotropic versions, respectively, for they are a 


formal discretization of the isotropic TV f, ,/ f2 + f?2dx and the anisotropic TV 


Jo(Scl + | A Dd x, respectively. The anisotropic TV is not rotational invariant; an 
image and its rotation can have a different TV value. Therefore, the discrete TV (3) 
deviates from the original isotropic TV. But being a piecewise linear function, some 
numerical techniques for quadratic and linear problems can be applied. Indeed, by 
introducing some auxiliary variables, the corresponding discrete ROF objective can 
be converted into a canonical quadratic programming problem [30]. 

Besides using finite difference approximations, a recent popular way is to 
represent TV on graphs [27]. To make the problem fully discrete, the range of the 
image is quantized to a finite set of K integers only, usually 0-255. The image is 
“leveled,” so that lie = | if the intensity of the (i, /)th pixel is at most k, and 


lig = 0 otherwise. Then, the TV is given by 


K-1 
Iflrv = > Oo wiise 


k=0 i,j St 


k k 
te ~ Jeo 


(4) 


where wj,;,5, iS a nonnegative weight. A simple choice is the four-connectivity 
model, where w;,j,5, = 1 if |i — s| + |j —¢| < 1 and w;j5, = O otherwise. 
In this case, it becomes the anisotropic TV (3). Different choices of the weights 
penalize edges in different orientations. 

A related concept introduced by Shen and Kang is the quantum total variation 
[55]. They studied the ROF model when the range of an image is a finite discrete set 
(preassigned or determined on the fly), but the image domain is a continuous one. 
The model is suitable for problems such as bar code scanning, image quantization, 
and image segmentation. An elegant analysis of the model and some stochastic 
gradient descent algorithms were presented there. 
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Nonlocal TV 

First proposed by Buades et al. [11], the nonlocal means algorithm renounces the 
use of local smoothness to denoise an image. Patches which are spatially far away 
but photometrically similar are also utilized in the estimation process — a paradigm 
which has been used in texture synthesis [28]. The denoising results are surprisingly 
good. Since then, the use of nonlocal information becomes increasingly popular. 
In particular, Bresson and Chan [10] and Gilboa and Osher [31] considered the 
nonlocal TV. The nonlocal gradient Vy, f for a pair of points x € Q andy € Q is 
defined by 


Vu_f (% yy) = Vw(x.y) (f (x) — f (y)). 


where w(x, y) is a nonnegative weight function which is presumably a similarity 
measure between a patch around x and a patch around y. As an illustration, a simple 
choice of the weight function is 


w (x,y) = ae POW 9? 4 que FOF? /o3 


where a; and o; are positive constants, and F(x) is a feature vector derived from 
a patch around x. The constants ~; may sometimes be defined to depend on x, so 
that the total weight over all y € Q is normalized to 1. In this case, the weight 
function is nonsymmetric with respective to its arguments. The first term in w is 
a measure of geometric similarity, so that nearby pixels have a higher weight. The 
second term is a measure of photometric similarity. The feature vector F can be the 
color histogram or any texture descriptor over a window around x. The norm of the 
nonlocal gradient at x is defined by 


Vue f GO| = i i [Var f @y)P ay, 


which adds up all the squared intensity variation relative to f(x), weighted by 
the similarity between the corresponding pair of patches. The nonlocal TV is then 
naturally defined by summing up all the norms of the nonlocal gradients over the 
image domain: 


/ Vai f| dx. 
Q 


Therefore, the nonlocal TV is small if, for each pair of similar patches, the intensity 
difference between their centers is small. An advantage of using the nonlocal TV 
to regularize images is its tendency to preserve highly repetitive patterns better. In 
practice, the weight function is often truncated to reduce the computation costs spent 
in handling the many less similar patches. 


1508 R. Chan et al. 


Further Applications 


Inpainting in Transformed Domains 

After the release of the image compression standard JPEG2000, images can be 
formatted and stored in terms of wavelet coefficients. For instance, in Acrobat 6.0 or 
later, users can opt to use JPEG2000 to compress embedded images in a PDF file. 
During the process of storing or transmission, some wavelet coefficients may be 
lost or corrupted. This prompts the need of restoring missing information in wavelet 
domains. The setup of the problem is as follows. Denote the standard orthogonal 
wavelet expansion of the images f and u by 


f@)=Yoajevjie), jeZkeZ, 


ik 
and 


u(B) =) > BieWin (x), fEeZkeZ, 


Jk 


where {W;,} is the wavelet basis, and {a;}, {Bj} are the wavelet coefficients of 
Ff and u given by 


ajk=(fiwjxe) and Bye = (ux), 


respectively, for 7 € Z, k € Z*. For convenience, u(B) is denoted by u when there 
is no ambiguity. Assume that the wavelet coefficients in the index set J are known, 
i.e., the available wavelet coefficients are given by 


Ajk> Gk) eT, 
fik = 0G 
, Uk) € Q\i. 
The aim of wavelet domain inpainting is to reconstruct the wavelet coefficients 
of u from the given coefficients &. It is well known that the inpainting problem 
is ill posed, i.e., it admits more than one solution. There are many different ways 
to fill in the missing coefficients, and therefore many different reconstructions in 
the pixel domain are possible. Regularization methods can be used to incorporate 
prior information about the reconstruction. In [23], Chan, Shen, and Zhou used TV 
to solve the wavelet inpainting problem, so that the missing coefficients are filled 
while preserving sharp edges in the pixel domain faithfully. More precisely, they 
considered the minimization of the following objective 


1 
F (B) = 5 90 xi (Ge — Bia) + Allallry GC) 
jk 


Numerical Methods and Applications in Total Variation Image Restoration 1509 


with xjx = lif (j, k) € I and xyjx = O if (j,k) € Q\I, and A is the 
regularization parameter. The first term in F is the data-fitting term, and the second 
is the TV regularization term. The method Chan, Shen, and Zhou used to optimize 
the objective is the standard gradient descent. The method is very robust, but it often 
slows down significantly before it converges. 

In [18], Chan, Wen, and Yip proposed an efficient optimization transfer algo- 
rithm to minimize the objective (5). An auxiliary variable ¢ is introduced to yield a 
new objective function: 


G66) = $* (In -91G + ee —61) +4 le Bllry 


where x denotes a diagonal matrix with diagonal entries x ;,,, and t is an arbitrary 
positive parameter. The function G is a quadratic majorizing function [43] of F. The 
method also has a flavor of the splitting methods introduced in section “Splitting 
Methods.” But a major difference is that the method here solves the original 
problem (5) without any alteration. It can be easily shown that 


F (p)= neG (¢, B) 


for any positive regularization parameter t. Thus, the minimization of G w.rt. 
(6, B) is equivalent to the minimization of F w.r.t. 8 for any t > O. Unlike the 
gradient descent method of [23], the optimization transfer algorithm avoids the use 
of derivatives of the TV. It also does not require smoothing out the TV to make it 
differentiable. The experimental results in [18] showed that the algorithm is very 
efficient and outperforms the gradient descent method. 


Superresolution 

Image superresolution refers to the process of increasing spatial resolution by fusing 
information from a sequence of low-resolution images of the same scene. The 
images are assumed to contain subpixel information (due to subpixel displacements 
or blurring), so that the superresolution is possible. 

In [24], Chan et al. proposed a unified TV model for superresolution imaging 
problems. They focused on the problem of reconstructing a high-resolution image 
from several decimated, blurred, and noisy low-resolution versions of the high- 
resolution image. They derived a low-resolution image formation model which 
allows multiple-shifted and blurred low-resolution image frames, so that it subsumes 
several well-known models. The model also allows an arbitrary pattern of missing 
pixels (in particular an arbitrary pattern of missing frames). The superresolution 
image reconstruction problem is formulated as an optimization problem which 
combines the image formation model and the TV inpainting model. In this method, 
TV minimization is used to suppress noise amplification, repair corrupted pixels 
in regions without missing pixels, and reconstruct intensity levels in regions with 
missing pixels. 
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Image Formation Model The observation model, Chan et al. considered, consists 
of various degradation processes. Assume that a number of m x n low-resolution 
frames are captured by an array of charge-coupled device (CCD) sensors. The goal 
is to reconstruct an Lm x Ln high-resolution image. Thus, the resolution is increased 
by a factor of L in each dimension. Let u be the ideal Lm x Ln high-resolution clean 
image. 

1. Formation of low-resolution frames. A low-resolution frame is given by 


DpqCu, 


where C is an averaging filter with window size L-by-L, and Dpg is the 
downsampling matrix which, starting at the (p, q)th pixel, samples every other 
L pixels in both dimensions to form an m x n image. 

2. Blurring of frames. This is modeled by 


Ay gDpgCu, 
where H,, is the blurring matrix for the (p, q)th frame. 
3. Concatenation of frames. The full set of L? frames are interlaced to form an 
mL x nL image: 


Au, 


where 


A=) DH ygD page. 
PY 


4. Additive Noise. 
Au+ yn, 


where each pixel in 7 is a Gaussian white noise. 
5. Missing pixels and missing frames. 


f =Ap(Au+y), 


where D denotes the set of missing pixels, and Ap is the downsampling matrix 
from the image domain to D. 

6. Multiple observations. Finally, multiple observations of the same scene, but with 
different noise and blurring, are allowed. This leads to the model 


fr =Ap, (Aputn) r=1,....R, (6) 
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where 


T 
A= YD, Hone Doge. 
Pq 


TV Superresolution Imaging Model To invert the degradation processes in (6), a 
Tikhonov-type regularization model has been used. It requires minimization of the 
following energy: 


R 
1 
Fu) = 5) MAv, Aru fell? +A llullry - 


r=1 


This model simultaneously performs denoising, deblurring, inpainting, and super- 
resolution reconstruction. Experimental results show that reasonably good recon- 
struction can be obtained even if five-sixth of the pixels are missing and the frames 
are blurred. 


Image Segmentation 
TV minimization problems also arise from image segmentation. When one seeks for 
a partition of the image into homogeneous segments, it is often helpful to regularize 
the shape of the segments. This can increase the robustness of the algorithm against 
noise and avoid spurious segments. It may also allow the selection of features 
of different scales. In the classical Mumford-Shah model [47], the regularization 
is done by minimizing the total length of the boundary of the segments. In this 
case, if one represents a segment by its characteristic function, then the length 
of its boundary is exactly the TV of the characteristic function. Therefore, the 
minimization of length becomes the minimization of TV of characteristic functions. 
Given an observed image f on an image domain Q, the piecewise constant 
Mumford-Shah model seeks a set of curves C and a set of constant ec = 
(Ci, C2,...,¢€L) which minimize the energy functional given by 


L 
F™S (Cc) = yy Lf (x) — cj] dx + B-Length(C). 
Q7 


1=1 


The curves in C partition the image into L mutually exclusive segments (2; for 
1 =1,2,..., L. The idea is to partition the image, so that the intensity of f in each 
segment (2; is well approximated by a constant c;. The goodness of fit is measured 
by the L* difference between f and c;. On the other hand, a minimum description 
length principle is employed which requires the curves C to be as short as possible. 
This increases the robustness to noise and avoids spurious segments. The parameter 
6B > O controls the trade-off between the goodness of fit and the length of the 
curves C. The Mumford-Shah objective is nontrivial to optimize especially when 
the curves need to be split and merged. Chan and Vese [24] proposed a level set- 
based method which can handle topological changes effectively. In the two-phase 
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version of this method, the curves are represented by the zero level set of a Lipschitz 
level set function ® defined on the image domain. The objective function then 
becomes 


FY .c1.02) =f HGO)LS @ —aF ds 
+f U-H@eL @-cl dx Bf IV). 
Q Q 


The function H is the Heaviside function defined by H(x) = lif x > 0, H(x) =0 
otherwise. In practice, we replace H by a smooth approximation H,, e.g., 


1 2 x 
Hi, (x) = 5 E + - arctan (*)| : 


Although this method makes splitting and merging of curves a simple matter, the 
energy functional is non-convex which possesses many local minima. These local 
minima may correspond to undesirable segmentations; see [45]. 

Interestingly, for fixed c; and cz, the above non-convex objective can be refor- 
mulated as a convex problem, so that a global minimum can be easily computed; 
see [22,56]. The globalized objective is given by 


FN (4, ¢1,¢) = —~aPre —olsu(x)d Vu 
were) = f {LF —aP Ef @) ase 


which is minimized over all u satisfying the bilateral constraints 0 < u < | and 
all scalars c; and c2. After a solution u is obtained, a global solution to the original 
two-phase Mumford-Shah objective can be obtained by thresholding u with 2 for 
almost every tt € [0,1], see [22,56]. Some other proposals for computing global 
solutions can be found in [45]. 

To optimize the globalized objective function (7), Chan et al. [22] proposed to 
use an exact penalty method to convert the bilaterally constrained problem to an 
unconstrained problem. Then the gradient descent method is applied. This method 
is very robust and easy to implement. Moreover, the exact penalty method treats the 
constraints gracefully, as if there is no constraint at all. But of course the gradient 
descent is not particular fast. 

In [42], Krishnan et al. considered the following discrete two-phase Mumford- 
Shah model: 


2 
u—-— 


2 


’ 


a 
FEN (u,c1,c2) = (8,u) + B llulley + 5 


where (-, -) is the /? inner product, s = (s;,;), and 


83 = (fig —0) (fig - 2) - 
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The variable u is bounded by the bilateral constraints 0 < u < 1. Whena = 0, 
this problem is convex but not strictly convex. When a > 0, this problem is 
strictly convex. The additive constant 5 is introduced in the third term so that 
the minimizer does not bias toward u = 0 or u = 1. This problem is exactly a 
TV denoising problem with bound constraints. Krishnan et al. proposed to use the 
primal-dual active-set method to solve the problem. Superlinear convergence has 
been established. 


Diffusion Tensors Images 

Recently, diffusion tensor imaging (DTI), a kind of magnetic resonance (MR) 
modality, becomes increasingly popular. It enables the study of anatomical struc- 
tures such as nerve fibers in human brains noninvasively. Moreover, the use of 
direction-sensitive acquisitions results in its lower signal-to-noise ratio compared to 
convectional MR. At each voxel in the imaging domain, the anisotropy of diffusion 
water molecules is interested. Such an anisotropy can be described by a diffusion 
tensor D, which is a 3 x 3 positive semi-definite matrix. By standard spectral theory 
results, D can be factorized into 


D=VAV', 


where V is an orthogonal matrix whose columns are the eigenvectors of D, and A is 
a diagonal matrix whose diagonal entries are the corresponding eigenvalues. These 
eigenvalues provide the diffusion rate along the three orthogonal directions defined 
by the eigenvectors. The goal is to estimate the matrix D (one at each voxel) from 
the data. Under the Stejskal-Tanner model, the measurement S;, from the imaging 
device and the diffusion tensor are related by 


Sk = Spe Ps P&, (8) 


where So is the baseline measurement, g; is the prescribed direction in which the 
measurement is done, and b > 0 is a scalar depending the strength of the magnetic 
field applied and the acquisition time. Since D has six degrees of freedom, six 
measurements at different orientations are needed to reconstruct D. In practice, 
the measurements are very noisy. Thus, matrix D obtained by directly solving (8) 
fork = 1,2,...,6 may not be positive semi-definite and is error-prone. It is thus 
often helpful to take more than six measurements and to use some least squares 
methods or regularization to obtain a robust estimate while preserving the positive 
semi-definiteness for physical correctness. 

In [60] Wang et al. and in [25] Christiansen et al. proposed an extension of the 
ROF to denoise tensor-valued data. Two major differences between the two works 
are that the former regularizes the Cholesky factor of D and uses a channel-by- 
channel TV regularization, whereas the latter regularizes the tensor D directly and 
uses a multichannel TV. 

The method in [25] is two staged. The first stage is to estimate the diffusion 
tensors from the raw data based on the Stejskal-Tanner model (8). The obtained 
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tensors are often noisy and may not be positive semi-definite. The next stage is to 
use the ROF model to denoise the tensor while restricting the results to be positive 
semi-definite. The trick they used to ensure positive semi-definiteness is very simple 
and practical. They observed that a symmetric matrix is positive semi-definite if and 
only if it has a Cholesky factorization of the form 


ei, 
where L is a lower triangular matrix 


ly, 0 0 
L =} In lyn 0 
131 132 133 


Then one can easily express D in terms of /;; for 1 < j <i <3: 


i Lilo Lyla 
D=D(L)=] Ilr 1,40, bails + lolx 
Tilsy lols: + lls 13, + 1 + 


The ROF problem, written in a continuous domain, is then formulated as 


min zo fleas an Df vai wl] | 


ij 


where D = (dij) is the observed noisy tensor field, and L is the unknown lower 
triangular matrix-valued function from Q to R°*°. Here, the matrix-valued version 
of TV is used. The objective is then differentiated w.r.t. the lower triangular part of 
L to obtain a system of six first-order optimality conditions. Once the optimal L 
is obtained, the tensor D can be formed by taking D = LL’ which is a positive 
semi-definite. 

The original ROF problem is strictly convex so that one can obtain the globally 
optimal solution. However, in this problem, due to the nonlinear change of variables 
from D to L, the problem becomes non-convex. But the authors of [25] reported 
that in their experiments, different initial data often resulted in the same solution, so 
that the non-convexity does not pose any significant difficulty to the optimization of 
the objective. 


4 Numerical Methods and Case Examples 


Fast numerical methods for TV minimization continue to be an active research 
area. Researchers from different fields have been bringing many fresh ideas to the 
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problem and led to many exciting results. Some categories of particular mention 
are dual/primal-dual methods, Bregman iterative methods, and graph cut methods. 
Many of these methods have a long history with a great deal of general theories 
developed. But when it comes to their application to the ROF model, many 
further properties and specialized refinements can be exploited to obtain even faster 
methods. Having said so, different algorithms may adopt different versions of TV. 
They have different properties and thus may be used for different purposes. Thus, 
some caution needs to be taken when one attempts to draw conclusions such as 
method A is faster than method B. Moreover, different methods have different 
degree of generality. Some methods can be extended directly to deblurring, while 
some can only be applied to denoising. (Of course, one can use an outer iteration 
to solve a deblurring problem by a sequence of denoising problems, so that any 
denoising algorithm can be used. But the convergence of the outer iteration has little, 
if not none, to do with the inner denoising algorithm.) This section surveys some 
recent methods for TV denoising and/or deblurring. The model considered here is a 
generalized ROF model which simultaneously performs denoising and deblurring. 
The objective function reads 


F (u) = 5 [ku praxea f iu, (9) 


where K is a blurring operator and A > 0 is the regularization parameter. For 
simplicity, we assume that K is invertible. When K is the identity operator, (9) 
is the ROF denoising model. 


Dual and Primal-Dual Methods 


The ROF objective is non-differentiable in flat regions where |Vu| = 0. This 
leads to much difficulty in the optimization process since gradient information 
(hence, Taylor’s expansion) becomes unreliable in predicting the function value 
even locally. Indeed, the staircase effects of TV minimization can introduce some 
flat regions which make the problem worse. Even if the standard procedure of 
replacing the TV with a reasonably smoothed version is used so that the objective 
becomes differentiable, the Euler-Lagrange equation for (9) is still very stiff to solve. 
Higher-order methods such as Newton’s methods often fail to work because higher- 
order derivatives are even less reliable. 

Due to the difficulty in optimizing the ROF objective directly, much recent 
research has been directed toward solving some reformulated versions. In particular, 
methods based on dual and primal-dual formulations have been shown to be very 
fast in practice. Actually, the dual problem (see (12) below) also has its own 
numerical difficulties to face, e.g., the objective is rank deficient and some extra 
work is needed to deal with the constraints. But the dual formulation brings many 
well-developed ideas and techniques from numerical optimization to bear on this 
problem. Primal-dual methods have also been studied to combine information from 
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the primal and dual solutions. Several successful dual and primal-dual methods are 
reviewed. 


Chan-Golub-Mulet’s Primal-Dual Method 

Some early work in dual and primal-dual methods for the ROF model can be found 
in [13,20]. In particular, Chan, Golub, and Mulet (CGM) [20] introduced a primal- 
dual system involving a primal variable u and a Fenchel dual variable p. It remains 
one of the most efficient methods today and is perhaps the most intuitive one. It is 
worthwhile to review it and see how it relates to the more recent methods. Their idea 
is to start with the Euler-Lagrange equation of (9): 


Vu 


K" Ku— K' f —Adiv | —————. | = 0. (10) 


V|Vul? +e 


Owing to the singularity of the third term, they introduced an auxiliary variable 


Vu 


a 
V|Vul +e 
py|Vul? +e = Vu 


K' Ku— K’ f —Adivp = 0. 


to form the system 


Thus, the blowup singularity is canceled. They proposed to solve this system by 
Newton’s method which is well known to converge quadratically locally if the 
Jacobian of the system is Lipschitz. Global convergence is observed when coupled 
with a simple Armijo line search [8]. The variable p is indeed the same as the 
Fenchel dual variable g in (1) when Vu 4 O and ¢ = O. Thus, p is a smoothed 
version of the dual variable g. Without the introduction of the dual variable, a direct 
application of the Newton’s method to the Euler-Lagrange equation (10) often fails 
to converge because of the small domain of convergence. 


Chambolle’s Dual Method 

A pure dual method is proposed by Chambolle in [14], where the ROF objective is 
written solely in terms of the dual variable. By the definition of TV in (1), it can be 
deduced using duality theory that 


fl 2 
int) 5 [ (Ku) dx+2f ivul 


1 
<=> inf sup sf (Ku- fPax+a f udivpdx| (11) 
« Ip (2 Jo Q 
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1 
<> sup int 5 | (Ku- fpdx+a f udivpds| 
Q Q 


42 
<> sup >| 
<i {2 Jo 


The resulting problem has a quadratic objective with quadratic constraints. In 
contrast, the primal objective is only piecewise smooth which is badly behaved 
when Vu = 0. Thus the dual objective function is very simple, but additional 
efforts are needed to handle the constraints. One can write down the Karush-Kuhn- 
Tucker (KKT) optimality system [8] of the discretized objective, which amounts 
to solving a nonlinear system of equations involving complementarity conditions 
and inequality constraints on the Lagrange multipliers. Interestingly, the Lagrange 
multipliers have a closed-form solution which greatly simplifies the problem. More 
precisely, the KKT system consists of the equations 


eae f 
K~"divp — + 
ivp 1 


2 
a ‘ (12) 


up = H (p) (13) 

1 (Ipi?-1) =0 (14) 
u>0 (15) 

IPI? <1, (16) 


where \t is the nonnegative Lagrange multiplier and 


H (p):=V [xT y divp — yk | 


Since 
|p| = | (p)|, 


if |p| = 1, then tj = |A(p)|; if |p| < 1, then the complementarity (14) implies 
ww = 0 and by (13) H(p) = 0, so that «1 is also equal to | H(p)|. This simplifies the 
KKT system into a nonlinear system of p only 


|H (p)|p = H (p). 
Chambolle proposes a simple semi-implicit scheme to solve the system: 


n+1 _ p” - tH (p”) 


Pp = ——__. 
p” + t|H (p")| 


Here, t is a positive parameter controlling the stepsize. The method is proven to be 
convergent for any 


ez ; (ary. (17) 
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where ||-|| is the spectral norm. This method is also faithful to the original ROF 
problem; it does not require approximating the TV by smoothing. 

The convergence rate of this method is at most linear, but for denoising problems, 
it usually converges fast (measured by the relative residual norm of the optimality 
condition) in the beginning but stagnates after some iterations (at a level several 
orders of magnitude higher than the machine epsilon). This is very typical for simple 
relaxation methods. Fortunately, visually good results (measured by the number of 
pixels having a gray level different from the optimal one after they are quantized 
to their 8-bit representation) are often achieved before the method stagnates [64]. 
However, when applied to deblurring, K is usually ill conditioned, so that the 
stepsize restriction (17) is too stringent. In this case, another outer iteration is often 
used in conjunction with the method; see the splitting methods in section “Splitting 
Methods.” 

Chambolle’s method has been successfully adapted to solve a variety of related 
image processing problems, e.g., the ROF with nonlocal TV [9], multichannel 
TV [10], and segmentation problems [4]. We remark that many other approaches 
for solving (12) have been proposed. A discussion of some first-order methods 
including projected gradient methods and Nesterov methods can be found in 
[3, 26, 61]. 


Primal-Dual Hybrid Gradient Method 

As mentioned in the beginning of Sect.4, the primal and dual problems have 

their own advantages and numerical difficulties to face. It is therefore tempting to 

combine the best of both. In [64], Zhu and Chan proposed the primal-dual hybrid 

gradient (PDHG) algorithm which alternates between primal and dual formulations. 
The method is based on the primal-dual formulation 


1 
G (u, p) := 5 [Kum pana f usivp dx — int sup; 


“ \pisi 


cf. formulation (11). By fixing its two variables one at a time, this saddle point 
formulation has two subproblems: 


sup G(u,p) and infG(u,p). 
Ipl<! - 


While one may obtain an optimal solution by solving the two subproblems to a 
high accuracy alternatively, the PDHG method applies only one step of gradient 
descent/ascent to each of the two subproblems alternatively. The rationale is that 
when neither of the two variables are optimal, there is little to gain by iterating each 
subproblem until convergence. Starting with an initial guess u°, the following two 
steps are repeated: 


pY*! = Ppj<i (p* — % Vu") 
uk*+! = uk — 6 (KT (Kuk — f) + Adivp*t!) 
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Here, Pipj<: is the projector onto the feasible set {p : |p| < 1}. The stepsizes 
t; and 9; can be chosen to optimize the performance. Some stepping strategies 
were presented in [64]. In [65], Zhu, Wright, and Chan studied a variety of stepping 
strategies for a related dual method. 

Numerical results in [64] show that this simple algorithm is faster than the split 
Bregman iteration (see section “Split Bregman Iteration”), which is faster than 
Chambolle’s semi-implicit dual method (see section “Chambolle’s Dual Method”). 
Some interesting connections between the PDHG algorithm and other algorithms 
such as proximal forward-backward splitting, alternating minimization, alternating 
direction method of multipliers, Douglas-Rachford splitting, split inexact Uzawa, 
and averaged gradient methods applied to different formulations of the ROF model 
are studied by Esser et al. in [29]. Such connections reveal some convergence theory 
of the PDHG algorithm in several important cases (special choices of the stepsizes) 
in a more general setting. 


Semi-smooth Newton's Method 

Given the dual problem, it is natural to consider other methods to solve its 
optimality conditions (13)-(16). A standard technique in optimization to handle 
complementarity and Lagrange multipliers is to combine them into a single equality 
constraint. Observe that the constraints a > 0, b > O and ab = O can be 
consolidated into the equality constraint 


o (a,b) := Va2+b?-a-b=0, (18) 


where ¢ is known as the Fischer-Burmeister function. Therefore, the KKT sys- 
tem (13)-(16) can be written as 


up = H (p) 
$(u.1-|pl’) =0. 


Ng et al. [48] observed that this system is semi-smooth and therefore proposed 
solving this system using a semi-smooth Newton’s method. In this method, if the 
Jacobian of the system is not defined in the classical sense due to the system’s 
lack of enough smoothness, then the Jacobian is replaced by a generalized Jacobian 
evaluated at a nearby point. It is proven that this method converges superlinearly 
if the system to solve is at least semi-smooth and if the generalized Jacobians 
at convergence satisfy some invertibility conditions. For the dual problem (12), 
the Newton’s equation may be singular. This problem is fixed by regularizing the 
Jacobian. 


Primal-Dual Active-Set Method 

Hintermiiller and Kunisch [37] considered the Fenchel dual approach to formulate 
a constrained quadratic dual problem and derived a very effective active-set method 
to handle the constraints. The method separates the variables into active and inactive 


1520 R. Chan et al. 


sets, so that they can be treated differently accordingly to their characteristics. They 
considered the case of anisotropic discrete TV norm (3), so that the dual variable 
is bilaterally constrained, i.e, —1 < p < 1, whereas the constraints in (12) are 
quadratic. In this setting, superlinear convergence can be established. 

To deal with the bilateral constraints on p, they proposed to use the primal-dual 
active-set (PDAS) algorithm. Consider the general quadratic problem, 


1 
min —(y,Ay)—(f,y), 
sn a Ad) Ney) 
where w is a given vector in R”. This problem includes (12) as a special instance. 
The KKT conditions are given by 


Ay = J, 
vO(w-y) =9, 
v>0, 


w-y =O, 


where v is a vector of Lagrange multipliers and © denotes the entrywise product. 
The idea of the PDAS algorithm is to predict the active variables A and inactive 
variables Z to speed up the determination of the final active and inactive variables. 
The prediction is done by comparing the closeness of v and w — y to zero. If wy — y 
is c times closer to zero than v does, then the variable is predicted as active. The 
PDAS algorithm is given by 


1. Initialize y°, v°. Set k = 0. 
2. Set Z* = {i : VF —c(w — y*); < 0} and A‘ = {i : vi —c(w — y*); > Of. 
3. Solve 
Ay*+! 4 pktl = f. 
yet = v on A‘, 
we =O on 2, 


4. Stop or set k = k + 1 and return to Step 2. 
Notice that the constraints a > 0, b > 0, and ab = 0 can be combined as a 
single equality constraint: 


min (a,cb) = 0 


for any positive constant c. Thus, the KKT system can be written as 


Ay+v =f, 
C (y, v) = 90, 
where C(y,v) = min(v,c(y¥ — y)) for an arbitrary positive constant c. The 


function C is piecewise linear, whereas the Fisher-Burmeister formulation (18) 
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is nonlinear. More importantly, applying Newton’s method (using a generalized 
derivative) to such a KKT system yields exactly the PDAS algorithm. This 
allows Hintermiiller et al. to explain the local superlinear convergence of the 
PDAS algorithm for a class of optimization problems that include the dual of 
the anisotropic TV deblurring problem [36]. In [37], some conditional global 
convergence results based on the properties of the blurring matrix K have also 
been derived. Their formulation is based on the anisotropic TV norm, and the 
dual problem requires an extra /? regularization term when a deblurring problem 
is solved. 


The dual problem (12) is rank deficient and does not have a unique solution in 
general. In [37], Hintermiiller and Kunisch proposed to add a regularization term, 
so that the solution is unique. The regularized objective function is 


[ \krtaivp-a-'pfaxty f |Pp|* dx, 
Q Q 


where P is the orthogonal projector onto the null space of the divergence operator 
div. Later in [38], Hintermiiller and Stadler showed that adding such a regularization 
term to the dual objective is equivalent to smoothing out the singularity of the TV in 
the primal objective. More precisely, the smoothed TV is given by fo ®(|V f|) d x, 
where 


s if|s| = y, 


®(s) = : 
a ie if |s| < y. 


An advantage of using this smoothed TV is that the staircase artifacts are reduced. 

In [41, 42], Krishnan et al. considered the TV deblurring problem with bound 
constraints on the image u. An algorithm, called nonnegatively constrained CGM, 
combining the CGM and the PDAS algorithms has been proposed. The image u and 
its dual p are treated as in the CGM method, whereas the bound constraints on u 
are treated as in the PDAS method. The resulting optimality conditions are shown 
to be semi-smooth. The scheme can also be interpreted as a semi-smooth quasi- 
Newton’s method and is proven to converge superlinearly. The method is formulated 
for isotropic TV, but it can also be applied to anisotropic TV after minor changes. 

However, Hintermiiller and Kunisch’s PDAS method [37] can only be applied to 
anisotropic TV because they used PDAS that can only handle linear constraints to 
treat the constraints on p. 


Bregman Iteration 
Original Bregman Iteration 


The Bregman iteration is proposed by Osher et al. in [49] for TV denoising. It has 
also been generalized to solving many convex inverse problems, e.g., [12]. In each 
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step, the signal removed in the previous step is added back. This is shown to alleviate 
the loss of contrast problem presented in the ROF model. Starting with the noisy 
image fo = f/f, the following steps are repeated for 7 = 0,1,2,...: 


1. Set 
_ {i 2 
uj+1 =argmin)— | (u— fj) dx+A] |Vul;. 
u 2 Jo Q 
2. Set fi4i = fj + Ff — js). 

In the particular case when f consists of a disk over a constant background, it 
can be proved that the loss of contrast can be totally recovered. Some theoretical 
analysis of the method can be found in [49]. 

For a general regularization functional J(u), the Bregman distance is defined as 


D5 (u,v) = J (u) — J (v) — (p,u—v), 


where p is an element of the subgradient of J. In case of TV denoising, J(u) = 
A Jo |Vul. Then, starting with fo = f, the Bregman iteration is given by 


1. Set 


1 
uj = argmin 5 f (u— fr dx+ Dd; (u, u;) 
Q 


u 


2. Set Siri = ti + (f —uj41). 
3. Set PjHi= Siri —f. 


In fact, steps 2 and 3 can be combined to pj41 = pj + f — uj+4i1 without 
the need of keeping track of f;. The above expression is for illustrating how the 
residual is added back to Fi: In this iteration, it has been shown that the Bregman 
distance between u; and the clean image is monotonically decreasing as long as 
the L>-distance is larger than the magnitude of the noise component. But if one 
iterates until convergence, then uj; — /, i.e., one just gets the noisy image back. 
This counterintuitive feature is indeed essential to solving other TV minimization 
problems, e.g., the basis pursuit problem presented next. 


The Basis Pursuit Problem 

An interesting feature of the Bregman iteration is that, in the discrete setting, if one 
replaces the term || — f'||? in the objective by ||Au — f||*, where Au = f is 
underdetermined, then upon convergence of the Bregman iterations, one obtains the 
solution of the following basis pursuit problem [63): 


min {J (u) |Au= f}. 
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When ||Au— /||* is used in the objective instead of ||u— f ||, the Bregman iteration 
is given by: 


1. Set 
1 
ujti = argmin} 5 ff (Au— fy dx+ Di (u, uj) 
u Q 


2. Set Si+i = ti + (f —Auj+1). 
3. Set pj41 = A? (fj4i— f). 


Split Bregman Iteration 

Recently, Goldstein and Osher [35] proposed the split Bregman iteration which can 
be applied to solve the ROF problem efficiently. The main idea is to introduce a 
new variable so that the TV minimization becomes an L! minimization problem 
which can be solved efficiently by the Bregman iteration. This departs from the 
original Bregman iteration which solves a sequence of ROF problems to improve the 
quality of the restored image by bringing back the loss signal. The original Bregman 
iteration is not iterated until convergence. Moreover, it assumes the availability of 
a basic ROF solver. The split Bregman method, on the other hand, is an iterative 
method whose iterates converge to the solution of the ROF problem. In this method, 
a new variable q = Vu is introduced into the objective function: 


min | > f w-fraxta | alas} (19) 
uq (2 Jo Q 


This problem is solved using a penalty method to enforce the constraint q = Vu. 
The objective with an added penalty is given by 


Gua =5 f ia-Vvubdx+ 5 fw frax+a f lala (20) 


Notice that if the variables (u, q) are denoted by y, then the above objective can be 
identified as 


min a) |Ay|? dx + 1h 
y 2 Q 
where 


Ay =q-—Vu, 
Jy) =$fou- fy dxta fo laldx’ 


This is exactly the basis pursuit problem when a@ —> oo. Actually, even with a fixed 
finite w, as mentioned in section “The Basis Pursuit Problem,” when the Bregman 
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iteration is used, it converges to the solution of the problem 


min {J (y) |Ay = 0}, 


so that the constraint q = Vu is satisfied at convergence. 
It is interesting to note that the split Bregman iteration can be viewed as a 
forward-backward splitting method [53]. Yet another point of view is provided next. 


Augmented Lagrangian Method 

In [62, 63], it is recognized that the split Bregman iteration is an augmented 
Lagrangian method [33]. This explains some good convergence behaviour of the 
split Bregman iteration. To motivate the augmented Lagrangian method, consider 
a general objective function J(u) with equality constraint H(u) = 0. The idea of 
penalty methods is to solve a sequence of unconstrained problems 


min | J) + GP 


with B — 0, so that the constraint H(u) = 0 is enforced asymptotically. However, 
one may run into the embarrassing situation where both H(u()) (where u(8) is the 
optimal u for a given 6) and B converge to zero in the limit. This could mean that the 
objective function is stiff when is very small. The idea of augmented Lagrangian 
methods is to use a fixed parameter. But the penalty term is added to the Lagrangian 
function, so that the resulting problem is equivalent to the original problem even 
without letting 8 > 0+. The augmented Lagrangian function is 


Heese noe gon: 
where v is a vector of Lagrange multipliers. Solving 4 a = aL = 0 fora 
saddle point yields exactly H(u) = O for any B > O. The Bregman iteration 
applied to the penalized objective (20) is indeed computing a saddle point of the 
augmented Lagrangian function of (19) rather than optimizing (20) itself. Therefore, 
the constraint Vu = q accompanied with (19) is exact even with a fixed a. 


Graph Cut Methods 


Recently, there is a burst of interest in graph cut methods for solving various 
variational problems. The promises of these methods are that they are fast for many 
practical problems and they can provide globally optimal solution even for “non- 
convex problems.” The discussion below is extracted from [15, 27]. Readers are 
referred to [15,27] and the references therein for a more thorough discussion of the 
subject. Since graph cut problems are combinatoric, the objective has to be cast in 
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a fully discrete way. That is, not only the image domain has to be discretized to a 
finite set but also the range of the intensity values. Therefore, in this framework, the 
given m-by-n image f is a function from Z,, x Z, to Zx. The ROF problem thus 
becomes 


m 


F (u) = Ie 3 Ui,j, — y+ Alfu ullry > min 


utZm XZn 7p 
27D yal 


where ||u||ry is a discrete TV (4). The next question is how to transform this 
problem to a graph cut problem in such a way that it can be solved efficiently. 
It turns out that the (fully discretized) ROF problem can be converted to a finite 
sequence of graph cut problems. This is due to the co-area formula which is unique 
to TV. Details are described next. 


Leveling the Objective 
Some notations and basic concepts are in place. For simplicity, the following 
discrete TV is adopted: 


m—\|n-—-1 


llullry = yy, |uisiyy — Mig] + [mij — ui, 


i=l j=l 


which is the ausolrOpt TV in (3), but with the range of u restricted to Zx. Recall 
that the binary image uv“ is defined such that each uk j equals Lifuj; < k and equals 
0 otherwise. Thus, it is the kth lower level set of u. Then the co-area formula states 
that the discrete TV can be written as 


K-2 
lellry = Do Ne ey 


k=0 


Thus, it reduces to the TV of each “layer’’. Note that the TV of the (K — 1)st level 
set must be zero, and therefore the above sum is only up to K — 2. 

The fitting term in the objective can also be treated similarly as follows. Notice 
that for any function g;_;(s), it holds that 


8ij (8) = S le eee 


> 
o 


~ 
i) 


an [gi 7&+1)- 8; (k)| Xk<s + 81; 0), 


> 
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where Xx<s = 1 if k < s and 0 otherwise. Define g;,;(k) = +(s — fi;)*. Then, 


(uj - fii) = Bij (ui,j) 
[gj (kK + 1) — 81,7 (KD) xXk<u,; + 81; O) 


II 
a> 
Ores 


ll 
iy iM 


[sig E+ D817 ©] (1 uh) + 817. 


As a result, the ROF objective can be expressed as 


K-2 


> [gi.7 &+D-8:,; ©] (1- ui i) +4 Je“nv p+: 


k=0 { ij 


where C = )/, ; 8:,; (0). 
By defining the objective function 


Ey \= > [aii (k+1)— 31; (k)] (1 —vh;) +A | [oy 


ig 


where v* is a binary function, the ROF problem is seen to be equivalent to 


subject to the inclusion constraints hg c= wee for alli, 7, k. The constraints make 


sure the binary functions {v"}, define the lower level sets of some function v. A 
very important result is that the minimization can be done independently for each 
v*; amazingly, the solutions {v*} satisfy the inclusion property automatically! See 


[27] for further details. 
Defining a Graph 


To minimize each F* wart. a aunty function v*, a graph cut method is used. First 
observe that since g;,;(k) = 5 Lik — fi. iV oF * can be simplified to 


F¥ (v) =D [5+ e-4,)] G-4)) +4 Ley: 


By absorbing some constants and dropping the superscript on v*, the objective takes 
the following form: 


1 
Fe@= (4, He ;) vig FA ll (21) 


i,j TV 
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Then, a graph with mn + 2 nodes is constructed in the following way: 


1. Each of the mn pixels is a node, labeled by (i, j) fori = 1,2,...,m and j = 
| ee 

2. Add two additional nodes, called the source S and the sink 7. 

3. For each (i, 7), connect it to (7 + 1, 7) and (i, 7 + 1) with capacity A. 

4. For each (i, /), connect S to it with capacity 5 +k— fi; if 5 +k—-fij >9 
and connect it to T with capacity fj; —k — 5 if fi; —k - 5 > 0. 


A cut (a.k.a. an sf-cut) in the graph is a partition (S,7) such that S € S and 
T € TJ. The cost of the cut C(S,7) is defined as the sum of the capacities of all 
edges from S to 7. For a given cut, let v;,; equals 1 if (i, 7) € S and equals 0 if 
(i, 7) € T. Then it can be verified that 


C(S.T) = Somax | 5 +k ~ fi .0h Vij + max | fi -k- 5.0} 
ij 
(l=wy) +4 “lev 


which is the same as F* in (21), up to the constant ¥;,; max { f;,; —k, 0}. Therefore, 
computing the minimum cut is equivalent to minimizing (21). It is also well known 
that the minimum cut problem is equivalent to the maximum flow problem. 

Recall that there are K — | graphs to cut. A simple way is to do them one by 
one using any classical maximum flow algorithm. But one can exploit the inclusion 
property to reduce the work; for instance, see the divide-and-conquer algorithm 
proposed in [27]. 

In graph cut methods, a fundamental question is what kind of optimization 
problems can be transformed to a graph cut problem. A particularly relevant 
question is whether a function is levelable, i.e., its minimization can be done by 
first solving the simpler problem on each of its level set, followed by assembling the 
resulting level sets. Interestingly, the only levelable convex regularization function 
(satisfying some very natural and mild conditions) is TV [27]. This indicates that 
TV is much more than just an ordinary semi-norm. 


Quadratic Programming 


The discrete anisotropic TV is a piecewise linear function. Fu et al. [30] showed 
that by introducing some auxiliary variables, one can transform the TV to a linear 
function but with some additional linear constraints. Together with the fitting term, 
the problem to solve has a quadratic objective function with linear constraints. 

The objective function considered is by Fu et al. 


1 
F(u)= 5 |Ku— fl’ +40 uit — u;,;| + |aez p41 —uj,j|, 


i,j 
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which can also be written as 
1 2 
F(u)= 5 |Ku— flo +A | Rul, 


where R is a 2mn-by-mn matrix. If the original isotropic TV is used, then it cannot 
be written in this form. 

The trick they used is to let v = Ru and then split it into positive and negative 
parts: vt = max(v,0) and v- = max(—v, 0). Then, the objective can be written as 


1 
G(u,vt,v-) = 5 iKu- f\P+a(itvt +17), 
which is a quadratic function. But some linear constraints are added: 
Fy. 


Ru=v 
vty > 0. 


Now, this problem can be solved by standard primal-dual interior-point methods. 
Here, “dual” refers to the Lagrange multipliers for the linear constraints. The major 
steps can be summarized as follows: 


1. Write down the KKT system of optimality conditions, which has a form of 
F(x, u,s) = 0, where x > 0 is the variable of the original problem (x = 
(u, vt, v~) in the present case); | is the Lagrange multipliers for the equality 
constraints; and s > 0 is the Lagrange multipliers for the inequality constraints. 

2. Relax the complementarity xs = 0 (part of f(x, w, s) = 0) toxs = v, where 

v>0. 

. Solve the relaxed problem f(x, |1,5) = 0 by Newton’s method. 

4. After each Newton’s iteration, reduce the value of v so that the solution of 
F(x, w, 8) = 0 is obtained at convergence. 


ww 


In this method, the relaxed complementarity xs = v forces the variables x, s 
to lie in the interior of the feasible region. Once the variables are away from the 
boundary, the problem becomes a nice unconstrained quadratic problem locally. 
The main challenge here is that the linear system to solve in each Newton’s iteration 
becomes increasingly ill conditioned. Under this framework, bound constraints 
such aS Umin < WU < Umax Or any linear equality constraints can be easily 
added. 


Second-Order Cone Programming 


The trick to “linearize” the TV presented in the last section does not work for 
isotropic TV. Goldfarb and Yin [34] proposed a second-order cone programming 
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(SOCP) formulation which works for the isotropic version (2). Moreover, its 
connection to SOCP allows the use of available SOCP solvers to obtain the 
solutions. The problem they considered is the constrained ROF problem: 


min [||| py 
u 
subject to 
llu— fll so, 


where o is the standard deviation of the noise which is assumed to be known. 
Let Wij = Uj+i,j — Wij and W; i = Uj j+i— Ui,;- The TV becomes 


2 > 
Dylon) +O) 
ij 
By introducing the variables v = f — u and t and the constraint 


x : y 2 7) 
(ws) + (w},) SU, 


the TV minimization problem becomes 


Sut. ut+v=f 


Wig = 4ijtl — Ui; 


(o,v) € cone™"t! 


(4.;,i;w?;) € cone’, 


Here, cone” is the second-order cone in R”: 
{x € R": ||(%2, %3,..-, Xn) || S xi}. 


The optimal solution satisfies 


so that 


a= Vw, +(w?)) = llallry. 
ij ij 


A SOCP formulation of the dual ROF problem is also given in [34]. 
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The SOCP can be solved by interior-point methods. The above formulation can 
be slightly simplified by eliminating u. But the number of variables (hence, the size 
of the Newton’s equation) is still several times larger than the original problem. 
Goldfarb and Yin proposed a domain decomposition method to split the large 
programming problem into smaller ones, so that each subproblem can be solved 
efficiently. Of course, the convergence rate of the method deteriorates as the domain 
is further split. 


Majorization-Minimization 


Majorization-minimization(MM) (or minorization-maximization) [43] is a well- 
studied technique in optimization. The main idea is that at each step of the method, 
the objective function is replaced by a simple one, called the surrogate function, 
such that its minimization is easy to carry out and the result gives a smaller objective 
value of the original problem. For a given objective, usually many surrogate 
functions are possible. In many cases, one can even reduce multidimensional 
problems into a set of one-dimensional problems. Methods of this class have 
been heavily used in statistics communities. Indeed expectation-maximization (EM) 
algorithms are special cases of MM. 

The use of MM to solving discrete TV problems can be traced back to the study 
of emission and transmission tomography reconstruction problems by Lange and 
Carson in [44]. Recently, some authors have applied the method to solving TV 
deblurring problems [6]. However, the method is actually the same as the classical 
lagged diffusivity fixed point iteration proposed by [58] for the particular surrogate 
function used in [6]. Nevertheless, it is still worthy to present the framework here 
because other surrogate functions can lead to different schemes. 

Denote by w* the kth iterate. In this method, the surrogate function (majorizer) 
O(u|u") is defined such that 


F (u‘) = O (uk \u*) 
F(iu <@ (u|u) , forallu 


Then, the next iterate is defined to be the minimizer of the surrogate function 
uk+! :— argmin QO (u\u‘) ; 
u 
In this way, the following monotonic decreasing property holds 


F a) <O (u'*" uk) <Q (u*|u*) =F (u‘) ; 


Presumably, the function Q should be chosen so that its minimum is easy to 
compute. In many applications, it may even be chosen to have a separable form 


O (uju*) = Oy (ui|u*) + Qo (u2|u*) + +++ + On (unlu*) 
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so that its minimization reduces to n’s one-dimensional (1D) problems. A promise 
of this method is that each iteration is very easy to carry out, which compensates its 
linear-only convergence. To construct a surrogate Ory for TV, first note that 


va= (v0) (4) <4 


for alla, b > 0. Let D, and Dy, be the forward difference operator in x and in y 
directions, respectively. Then, 


+ (Dy uij) 


2 
+ (Dyuk;) + 5 (Dy Wij) +(Dy Wij) 


ij (Deuk )+(Dyut,) 


| 
[4] 
= 
S 
= 


lWllry = 


IA 
Nie 
=M 
anes 
o— 
S 
ro) 
“I 
oF 
— 


The surrogate is thus defined as 


i Wea) (Dyas) 


Orv (ulu*) = = un) + 
fa a 257 Tout.) (Dat) 


which is quadratic in u. Notice that the 2D discrete gradient matrix is given by 


Vin © In 
V= 
oak 


where V,, is the m-by-m1D forward difference matrix (under Neumann boundary 
conditions) 


-1 1 


Let AK, =1/y( Dy a) (Dut a and let 


AF = diag (Af)... ae Meas ple )s 


° 9 “m,n? mn 


The surrogate becomes 


1 
Ory (uh) = 5 fe py + 50 VA Ve. 
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In this case, the minimization of Ory cannot be reduced to a set of 1D problems. 
But it does become quadratic. 
Finally, the majorizer for the ROF model is 


O (ula) = 5 1Ku— FIP + AQry (ule). 


While this method completely bypasses the need to optimize the TV term directly, 
each iteration requires solving the linear system 


(K™K +AVTAKV) uit! = KT f. 


This scheme is exactly the lagged diffusivity fixed point iteration. Assume that 
K is full rank, then the linear system is positive definite. A standard way is to 
use preconditioned conjugate gradient to solve. Many preconditioners have been 
proposed for this problem in the 1990s, e.g., cosine transform and multigrid and 
multiplicative operator splitting; see [17] and the references therein. However, due 
to the highly varying coefficients in A¥, it can be nontrivial to solve efficiently. 


Splitting Methods 


Recently, there have been several proposals for solving TV deblurring problems 
based on the idea of separating the deblurring process and the TV regularization 
process. Many of them are based on the idea that the minimization of an objective 
of the form 


F (u) = Ji (u) + Jn (A), 


with Aa linear operator, can be approximated by the minimization of either of the 
following two objectives: 


G (u,v) = Ji (u) + & |lu—v|? + J (Av), 
G (u,v) = Jy (u) + & |Au-vI? + 0), 


where « is a large scalar. Then G is minimized w.r.t. uv and v alternatively. In this 
way, at each iteration, the minimization of J; and J is done separately. The same 
idea can be generalized to split an objective with n terms to an objective with n 
variables. 

Consider the discrete ROF model: 


1 
Fu) = 5 (Ku fl +A lVallh. 


Huang et al. [39] and Bresson and Chan [10] considered the splitting 
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1 a 
G u,v) = 5 [Ku fl + 5 lev? + A lllVollh 
In this case, the minimization w.r.t. u becomes 
(K’K+al)u=K'f +a, 


which can be solved with the fast Fourier transform (FFT) in O(N log NV) operations 
when the blurring matrix K can be diagonalized by a fast transform matrix. The 
minimization w.r.t. v is the ROF denoising problem which can be solved using any 
of the aforementioned denoising method. Both [39] and [10] employed Chambolle’s 
dual algorithm. The point is that solving TV denoising is much easier than solving 
TV deblurring (directly). Moreover, some algorithms such as those based on graph 
cut cannot be applied to deblurring directly. The reason is that the pixel values in the 
fitting are no longer separable, which in turn makes the fitting term not “levelable.” 
However, using the splitting technique, one can now apply graph cut methods to 
solve each denoising problem. 

This method is generally very fast. Moreover, it often works for a large range 
of a. But when aq is too large, the Chambolle’s iteration may slow down. This 
splitting method has also been applied to other image processing problems such 
as segmentation [10]. 

An alternative splitting is proposed by Wang et al. [59]. The bivariate function 
they used is given by 


1 a 
Gv) = 5 (Ku fll + 5 ilVe— vl? +A lle 
The minimization w.r.t. u requires solving 
(K'K —aA) u=K' f+a, 


where A is the 2D Laplacian. This equation can again be solved with FFT in 
O(N log N) operations. The minimization w.r.t. vgs decoupled into N minimiza- 
tion problems (one for each pixel) of two variables. A simple closed-form solution 
for the 2D minimization problems is available. Therefore, the computation cost per 
iteration is even less than the approach taken in [39] and [10]. Remark that this 
objective is indeed the same as the split Bregman method (20). A difference is that 
when the split Bregman iteration converges, it holds exactly that Vu = v. But the 
simple alternating minimization used in most splitting methods does not guarantee 
Vu = v at convergence. 

An alternative splitting is introduced by Bect et al. in [5]. It is based on the 
observation that, for any symmetric positive definite matrix B with ||B|| < 1, it 
holds that 


te _— yy? 
(By.y) = min {[lu— v7 + (Cu.a)} 
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for all v € R“, where C = B(J — B)~!. Then, the ROF model can be formulated 
as the minimization of the following bivariate function: 


G (u.9) = 5 (Iu VP + (Cun) + 5 (IFIP 21K f)) +A NIV 


where jt > 0 such that jt||K’ K|| < 1 and B = ,.K’ K. The minimization of G 
w.r.t. uhas a closed-form solution u = (I—B)v = (I—.K? K)v. The minimization 
of G w.r.t. v is a TV denoising problem. At convergence, the minimizer of F is 
exactly recovered. An interesting property of this splitting is that it does not involve 
any matrix inversion in the alternating minimization of G. 


5 Conclusion 


In this chapter, some recent developments of numerical methods for TV minimiza- 
tion and their applications are reviewed. The chosen topics only reflect the interest 
of the authors and are by no means comprehensive. It is also hoped that this chapter 
can serve as a guide to recent literature on some of these recent developments. 
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Abstract 


This chapter presents an overview of the Mumford and Shah model for image 
segmentation. It discusses its various formulations, some of its properties, the 
mathematical framework, and several approximations. It also presents numerical 
algorithms and segmentation results using the Ambrosio-Tortorelli phase-field 
approximations on one hand and level set formulations on the other hand. 
Several applications of the Mumford-Shah problem to image restoration are also 
presented. 


1 Introduction 


An important problem in image analysis and computer vision is the segmentation 
one, aiming to partition a given image into its constituent objects, or to find 
boundaries of such objects. This chapter is devoted to the description, analysis, 
approximations, and applications of the classical Mumford and Shah functional 
proposed for image segmentation. In [61-63], David Mumford and Jayant Shah 
have formulated an energy minimization problem that allows to compute optimal 
piecewise-smooth or piecewise-constant approximations u of a given initial image 
g. Since then, their model has been analyzed and considered in depth by many 
authors by studying properties of minimizers, approximations, and applications to 
image segmentation, image partition, image restoration, and more generally image 
analysis and computer vision. 

Let Q C R¢ be the image domain (an interval if d = 1; a rectangle in the plane 
if d = 2; or a rectangular parallelepiped if d = 3). More generally, it is assumed 
that Q is open, bounded, and connected. Let g:2 — R be a given grayscale image 
(a signal in one dimension, a planar image in two dimensions, or a volumetric image 
in three dimensions). It is natural and without losing any generality to assume that 
g is a bounded function in Q, thus g € L™ (Q). 

As formulated by Mumford and Shah [63], the segmentation problem in image 
analysis and computer vision consists in computing a decomposition 


Q=Q,UR,ZU...UQ, UK 
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of the domain of the image g such that 


1. The image g varies smoothly and/or slowly within each Q;. 
2. The image g varies discontinuously and/or rapidly across most of the boundary 
K between different Q;. 


From the point of view of approximation theory, the segmentation problem may 
be restated as seeking ways to define and compute optimal approximations of a 
general function g(x) by piecewise-smooth functions u(x), i.e., functions u whose 
restrictions u; to the pieces Q; of a decomposition of the domain Q are continuous 
or differentiable. 

In what follows, &2; will be disjoint connected open subsets of a domain Q, each 
one with a piecewise-smooth boundary, and K will be a closed set, as the union of 
boundaries of Q; inside Q, thus 


Q=Q,URZU...UQ,UK, K=QN(0Q,U...U0Q,). 


The functional E to be minimized for image segmentation is defined by [61-63], 
E(u, K) = | (u— g)dx +f |Vul? dx + v|K|, (1) 
Q Q/K 


where u : Q —> R is continuous or even differentiable inside each Q; (or u € H! 
(Q;)) and may be discontinuous across K. Here, |K| stands for the total surface 
measure of the hypersurface K (the counting measure if d = 1, the length measure 
if d = 2, the area measure if d = 3). Later, |K| will be defined by H?—!(K), the 
d — 1 dimensional Hausdorff measure in R¢. 

As explained by Mumford and Shah, dropping any of these three terms in (1), inf 
E = 0: without the first, take u = 0, K = 0; without the second, take u = g, K = 
0; without the third, take, for example, in the discrete case K to be the boundary of 
all pixels of the image g, each Q; be a pixel and u to be the average (value) of g 
over each pixel. The presence of all three terms leads to nontrivial solutions u, and 
an optimal pair (u, K) can be seen as a cartoon of the actual image g, providing a 
simplification of g. 

An important particular case is obtained when F is restricted to piecewise- 
constant functions u, i.e., u = constant c; on each open set 02;. Multiplying E by 
pL? gives 


wu.) => | (g —ci)°dx + vo|K|, 
i Jo; 


where v9 = v/j?. It is easy to verify that this is minimized in the variables c; by 
setting 


So; g(x)dx 


c; = meang,(g) = 12] 
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where |Q;| denotes here the Lebesgue measure of Q; (e.g., areaif d = 2, volume if 
d = 3), so it is sufficient to minimize 


Eo(K) = De [ (g — meang, g)’dx + vo|K|. 


It is possible to interpret Eo as the limit functional of E as 4 — 0 [63]. 
Finally, the Mumford and Shah model can also be seen as a deterministic 
refinement of Geman and Geman’s image restoration model [41]. 


2 Background: The First Variation 


In order to better understand, analyze, and use the minimization problem (1), it is 
useful to compute its first variation with respect to each of the unknowns. 

The definition of Sobolev functions u € W!*(U) [1] is now recalled, necessary 
to properly define a minimizer u when K is fixed. 


Definition 1. Let U C R¢ be an open set. Let W!?(U) (or H'!(U)) denote the set 
of functions u € L*(Q), whose first-order distributional partial derivatives belong 
to L?(U). This means that there are functions u;,..., uq € L?(U) such that 


0 
/ u(x) ¢ (x)dx = -| uj (x) (x)dx 
U OX; U 
for 1 <i <d and for all functions ¢ € C>° (U). 
Let ae denote the distributional derivative u; of u and Vu = (#, ees 7) its 
distributional gradient. In what follows, |Vu|(x) denotes the Euclidean norm of the 
gradient vector at x.H!(U) = W!?(U) becomes a Banach space endowed with the 


norm 
1/2 
_ > d du \* 
lula) = [fewer f (2) dx}. 


Minimizing in u with K Fixed 


K is assumed to be fixed, as a closed subset of the open and bounded set Q C R¢, 
and denote by 


E(u) = ef (u— axe [ |Vul? dx, 
Q/K Q/K 
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for u € W!?(Q\K), where Q\K is open and bounded, and g € L7(Q\K). The 
following classical results are obtained as consequences of the standard method of 
calculus of variations. 


Proposition 1. There is a unique minimizer of the problem 


inf E(u). (2) 
ueW!2(Q/K) 


Proof ([38]). First, note that 0 < inf E < +00, since choosing up = 0, then 
E(uo) = We forge 8° (x)dx < +00. Thus, let m = inf,E(u) and {u;}j>1 € 


W'?(Q\K) be a minimizing sequence such that limj—o0E(u;) = m. 


Recall that for u, v € L?, 


2 
u+yv u—vqj2 > 1 ) 
= + = |Nvll2. 
>|, trl, = zal + 2 ve 
and so 
u+v|? 2 2 u—vil2 
=| = 5 lui +50Mb-|--[,. 3) 


Let u, v € W!7(Q\K), thus E(u), E(v) < oo, and apply (3) to u — g and v — g, 
and then to Vu and Vv; then 


E (**) = 7£W) + 3£() - & Joye Ju —v)? dx — 4 JoyK |V(u—v)? dx 
2 2 
7 ie vIhir.2¢@a/x) + (1 - i) Vw — vB if i> i 


= EW) + 3EO)— 9 ; a 
ze — vilivi2~ayxy + (4 - 1) je v[2it <= 


(4) 
Choosing u, v € W!7(Q\K), such that E(u), E(v) < m +e, then 
m < E(*) <m+e- 
2 2 
& lu — vllavsagayxy + (1— &) IV — VDI ify = 
tlle viFaayey + (4 = 1) lle v3 iff < 
thus, 
4e sel pe 
2 ifs 2 
u—v\|lq. <¢# ; (5) 
| llwi2@ayK) ee eH 
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Let w; = uj — uy. From (5), {w;} is a Cauchy sequence in W!?(Q\K)); let w 
denote its limit and set uy = u; + w. Then, 


E(u) = 2? |luo — gllz + Vuolls = 2? a — g) + wis + [Vin + Voll 


= limj++e0 [He | — 9) +; I; + || Var + Vw; [2] 


= lim j++00 E(uj;) =m, 


by the continuity of L? norms. This shows the existence of minimizers. The 
uniqueness follows from (5) by taking e = 0. 


Proposition 2. The unique solution u of (2) is solution of the elliptic problem 


i Vu(x)- Vv(x)dx = - f [u(x) — g(x)] v(x)dx, Vv € W'"(Q/K), 
Q/K Q/K 6 


or of 


Au = ?(u-g) 


in the sense of distributions in Q\K, with associated boundary condition Al =0 


on 0(Q\ K), where N is the exterior unit normal to the boundary. 


Proof. Indeed, let ¢ +> A(e) = E(u+ev) for f € Rand arbitrary v € W!?(Q\K). 
Then, A is a quadratic function of ¢, given by 


A(e) = pe? Josx U- g)ydx + ey? Jork vdx + ep? Joye (u- g)vdx 
+ fos |Vul’ dx + 2 for \Vvl? dx + 26 fo). Vu: Vvdx, 
with 
A'(s) = Zep? Sark vdx + 2p? Soy (u— g)vdx + 2€ fork |Vv|° dx 
+2 fork Vu-Vvdx, 


and 


A’(0) = 2? (u— g)vdx + 2 | Vu-Vvdx, 
Q/K Q/K 


Since E(u) = A(O) < A(e) = E(u ev) forall f € Randall v € W!7(Q\K), it 
implies that A/(0) = 0 for all such v, which yields the weak formulation (6). 
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If in addition u would be a strong classical solution of the problem, or if it would 
belong to W2?(Q\ K), then integrating by parts in the last relation gives 


A’(0) = au? f (u— g)vdx + 2 | (Vu)vdx + 2 | Vu-Nvdx =0 
Q/K Q/K a(Q/K) 


Taking now v € C)(Q\K) C W!7(Q\K) gives 
Au = w?(u— g)inQ/K. 


Using this and taking now v € C!(Q\K), the associated implicit boundary 
condition Vu.N = a = 0 on the boundary of Q\K (in other words, on the 
boundary of Q and of each Q;) is obtained. 

Assume now that g € L® (Q2\K), which is not a restrictive assumption when 
g represents an image. It can be shown that the unique minimizer u of (2) satisfies 
Ilulloo < ||g\loo (as expected, due to the smoothing properties of the energy). To 
prove this, first the following classical lemma is stated (see, e.g., Ref. [38], Chapter 
A3). 


Lemma 1. /f Q\K is open, and if u € W!7(Q\K), then ut = max(u,0) also 
lies in W!?(Q\K) and |Vut (x)| < |Vu (x)| almost everywhere. 


Now let u* (x) = max{ —||g|]oo, min(||g||oo, u(x))} be the obvious truncation 
of u. Lemma | implies that u* € W!?(Q\K) and that JoyK |Vu*(x)|? dx < 
JovK |Vu(x)|? dx. Obviously, also Jo\n (u* — gy dx < Sayn u- g)* dx, thus 
E(u*) < E(u). But wis the unique minimizer of £; therefore, u(x) = u* (x) almost 
everywhere and ||u|loo < ||g||oo- 


Remark I. Several classical regularity results for a weak solution u of (2) can be 
stated: 


° Ifg € L© (Q\K), then u € C). (Q\K) (see e.g., Ref. [38], Chapter A3). 
* If g € L7(Q\K), then u € W2?(Q\K) = H2.(Q\K), which implies that wu 


solves the PDE (see, e.g., Ref. [39], Chapter 6.3). 


Au = w(u— g)ae. inQ/K. 


3 Minimizing in K 


Here we formally compute the first variation of E(u, K) with respect to K. Let us 
assume that (u, K) is a minimizer of E from (1), and we vary K. Let us assume that 
locally, K NU is the graph of a regular function ¢, where U is a small neighborhood 
near a regular, simple point P of K. 
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Without loss of generality, it can be assumed that U = D x I where J is an 
interval in Rand KNU = {(x1, %2,...,x¢4) €U = DxI: xg = O(%1,...,Xa-1)}. 
Let u* denote the restriction of u to 


Ut = {(%1, X2,...,Xa) 1 Xa > O(%,...,Xa-f OU, 
and u” the restriction of u to 
U~ = {(x1,X2,...,Xd) 1 Xa > O(%1,...,Xa-1)} NU, 


and choose H'! extensions of ut from U* to U, and of u~ from U~ to U. For small 
€, define a deformation K, of K inside U as the graph of 


Xa = b(m1,...,Xd-1) t EW(M1,...,Xd-1), 


such that y is regular and zero outside D, and K, = K outside U. Define 


u(x) if x EU, 
u,(x) = ¢ (extension of u*)(x) if x € U,x above K, NU 
(extension of u~)(x) if x € U,x above K,N U. 


Now, using z = (X],..-, Xg-1), 


E(u., K,) — E(u, K) = pw? fy, [Ue — 8)’dx — (u— g)"] dx 
+ fujn, \Wuel? dx = fyjx |\Vul? dx + v [|KeNU|-|K OUI] 
=) 2 f, Ugs b(2)+ey(2) [(u- <3) =r = aa dxa) Ge 
+ fy ( a [Iver P = |vut|?] dxa) dz 


+vfp lv +1Vd+ewP- V1 + vor | dz 


Thus, 


lim, 9 “#24924 = yy? fi, [CW - 8)? — ut - 8))] |xseaeq¥ @dz 


zs Vo:V 
+ fp [IVP = [Var | 1e so Ode + fy Feeds = 


for all such w, since (u, K) is a minimizer. Integrating by parts, we formally obtain 
forall w: 


In {Leer ae) Hime) = (tut =e) Iver’) | |=90 


—v div aa w(2dz = 0, 
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and obtain the first variation with respect to K, 


[ar —g)+ [vu P]—[w2ewr —gy+ [vu |? |-vaiv eae 0 


Vi+lVel? 
(7) 


on K M U. Noticing that the last term represents the curvature of K M U, and if we 
write the energy density as 


e(u; x) = p?(u(x) — g(x))? + |Vu(x)/*, 


we finally obtain 


e(ut) —e(u_) + vcurv(K) = 0 onK 


(at regular points of K, provided that the traces of u and of |Vu| on each side of K 
are taken in the sense of Sobolev traces). 

This section is concluded by stating another important result from [63] regarding 
the type of singular points of K, when (u, K) is a minimizer of F from (1), in two 
dimensions, d = 2. For the rather technical proof of this result, the reader is referred 
to the instructive and inspiring constructions from [63]. 


Theorem 1. Let d = 2. If(u, K) is a minimizer of E(u, K) such that K is a union of 
simple C':!-curves K; meeting 0Q and meeting each other only at their endpoints, 
then the only vertices of K are: 


1. Points P on the boundary 0&2 where one K; meets 0Q perpendicularly 
2. Triple points P where three K; meet with angles 21/3 
3. Crack tips where a K; ends and meets nothing. 


The later sections will discuss cases when the minimizer wu is restricted to a 
specific class of piecewise-constant or piecewise-smooth functions. 


4 Mathematical Modeling and Analysis: The Weak 
Formulation of the Mumford and Shah Functional 


To better study the mathematical properties of the Mumford and Shah functional 
(1), it is necessary to define the measure of K as its d— 1-dimensional Hausdorff 
measure H¢—!(K), which is the most natural way to extend the notion of length to 
nonsmooth sets. Recall the definition of the Hausdorff measure [4, 38, 40]. 


Definition 2. For K C R¢@ andn > 0, set 


H"(K) = sup H2(K), 
e>0 
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called the n-dimensional Hausdorff measure of the set K, where 


Hi(K) = cyint {| (diam 4))"\. 


oo 
i=1 


where the infimum is taken over all countable families { A; } 
that 


of open sets A; such 


K Cyi=! A; and diam A; < e forall i. 


Here, the constant c, is chosen so that #” coincides with the Lebesgue measure on 
n planes. 


Remark 2. When n is an integer and K is contained in a C!-surface of dimension 
n, H" (K) coincides with its n-dimensional surface measure. 
Consider a first variant of the functional, 


E(u, K) = ye f (u—g)’dx + i |Vul? dx + vH4-!(K). (8) 
Q/K Q/K 


In order to apply the direct method of calculus of variations for proving existence of 
minimizers, it is necessary to find a topology for which the functional is lower semi- 
continuous, while ensuring compactness of minimizing sequences. Unfortunately, 
the last functional K +> H4~!(K) is not lower semicontinuous with respect to any 
compact topology [4, 8, 38]. To overcome this difficulty, the set K is substituted 
by the jump set S, of u, thus K is eliminated, and the problem, called the weak 
formulation, becomes, in its second variant, 


int F(u) = tal (u—g)?dx + / |Vul? dx + rat's (9) 
u Q/S, Q/Sy 


For illustration, also given is the weak formulation in one dimension, for signals. 
The problem of reconstructing and segmenting a signal u from a degraded input g 
deriving from a distorted transmission can be modeled as finding the minimum 


inf 
u 


b 
wf ugar f |u’| dt + vA(S,) . 
a (a,b)/Sy 


where Q = (a, b), S,, denotes the set of discontinuity points of u in the interval (a, 
b), and #(S,) = °(S,,) denotes the counting measure of S, or its cardinal. 

In order to show that (9) has a solution, the following notion of special functions 
of bounded variation and the following important lemma due to Ambrosio [3,4] are 
necessary. 
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Definition 3. A function u € L'(Q) is a special function of bounded variation on 
Q if its distributional derivative can be written as 


Du = Vudx + (ut — u-)Ny |. 


such that Vu € L'!(Q), S, is of finite Hausdorff measure, (ut — u-) Nuys, € 
L! (Q HI |s,,.R“), where u* and u~ are the traces of u on each side of the jump 


part S,, and Ny is the unit normal to S,. The space of special functions of bounded 
variation is denoted by SBV(&). 


Lemma 2. Let uw, € SBV(Q) be a sequence of functions such that there 
exists a constant C > O with |u(x)| < C < waex € Q and 
ie |Vun|- dx+H4"!(Su,) < C. Then, there exists a subsequence unk converging 
a.e. to a function u € SBV(Q). Moreover, Vunx converges weakly in L?(Q)4 to Vu, 
and 


Ho S,) Slim inky, e591? (Sis, ): 


Theorem 2. Let g € L® (Q), with Q C R¢ open, bounded, and connected. There 
is a minimizer u € SBV(Q) NL™ (Q) of 


F(u) = wf (u— g)’dx +f |Vul* dx + vHT"(S,). 


u 


Proof. Notice that 0 < infsBviayn(@) < oo, because we can take up = 0 € 
SBV(Q) NL® (Q) and using the fact that g € L®(Q) C L?(Q), F(uo) < 
oo. Thus, there is a minimizing sequence u, € SBV(Q) NL (Q) satisfying 
lim,—+o0F(u,) = infF. Also notice that, by the truncation argument from before, 
it can be assumed that ||un|loo < ||glloo < oo. Since F(u,) < C < o for 
all n > 0, and using g € L®(Q) C L?(Q), we deduce that ||u,||2 < C and 
Joys, |Vun le dx+H4"!(S,,) < C for some positive real constant C. Using these 
and Ambrosio’s compactness result, it can be deduced that there is a subsequence 
Unk Of Un, and u € SBV(Q), such that u,, + win L?(Q), Vun, > Vuin L2(Q)¢. 
Therefore, F(u) <lim inf,p+o0F (unx~) = inf F, and it can also be deduced that 
Ill]oo < |1£ loo. 


For additional existence, regularity results and fine properties of minimizers, 
and for the connections between problems (8) and (9), the reader is referred to 
Dal Maso et al. [55,56], the important monographs by Morel and Solimini [60], 
by Chambolle [26], by Ambrosio et al. [4], by David [38], and by Braides [19]. 
Existence and regularity of minimizers for the piecewise-constant case can be found 
in [63], Congedo and Tamanini [52, 57, 75, 76], and Larsen [51], among other 
works. 
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5 Numerical Methods: Approximations to the Mumford and 
Shah Functional 


Since the original Mumford and Shah functional (1) (or its weak formulation 
(9)) is nonconvex, it has an unknown set K of lower dimension and is not the 
lower-semicontinuous envelope of a more amenable functional, and it is difficult 
to find smooth approximations and to solve the minimization in practice. Several 
approximations have been proposed, including the weak membrane model and 
the graduate non-convexity of Blake and Zisserman [16] (which can be seen 
as a discrete version of the Mumford and Shah segmentation problem); discrete 
finite differences approximations starting with the work of Chambolle [23-25] 
(also proving the I’-convergence of Blake-Zisserman approximations to the weak 
Mumford-Shah functional in one dimension); finite element approximations by 
Chambolle and Dal Maso [27] and by Chambolle and Bourdin [17, 18]; phase-field 
elliptic approximations due to Ambrosio and Tortorelli [5,6] (with generalizations 
presented by [19] and extensions by Shah [74], and Alicandro et al. [2]); region 
growing and merging methods proposed by Koepfler et al. [48], by Morel and 
Solimini [60], and by Dal Maso et al. [55,56] and level set approximations proposed 
by Chan and Vese [29-32, 79], by Samson et al. [71], and by Tsai et al. [78]; 
and approximations by nonlocal functionals by Braides and Dal Maso [20], among 
other approximations. Presented in this section in much more detail are the phase- 
field elliptic approximations and the level set approximations together with their 
applications. 

For proving the convergence of some of these approximations to the Mumford 
and Shah functional, the notion of -convergence is used, which is briefly recalled 
below. The interested reader is referred to Dal Maso [54] for a comprehensive 
introduction to ’-convergence. 

The reader is referred to the monographs and textbooks by Braides [19], by 
Morel and Solimini [60], and by Ambrosio et al. [4] on detailed presentations of 
approximations to the Mumford and Shah functional. 


Definition 4. Let X = (X, D) be a metric space. We say that a sequence Fj: X > 
[—oo, +00] I'-converges to F: X — [—oo, +00] (as j — oo) if for all u © X we 


have 


1. (lim inf inequality) for every sequence (u;) C X converging to u, 
F(u) < lim inf; F; (u;) (10) 


2. (existence of a recovery sequence) there exists a sequence (u;) C X converging 
to u such that 


F(u) = lim sup; Fj (uj), 
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or, equivalently by (10), 
Fu) = lim ; Fj (u;). 


The function F' is called the I-limit of (7; ) (with respect to D), and we write 
F= T-lim; F; . 


The following fundamental theorem is essential in the convergence of some of 
the approximations. 


Theorem 3 (Fundamental Theorem of I’-convergence). Let us suppose that 
F =T-lim; F;, and let a compact set C C X exist such that infy F; = infc F; 
for all j. Then, there is minimum of F over X such that 


min F = liminf F;, 
X j Xx 


and if (uj) C X is a converging sequence such that lim; F; (u;) = lim; infy Fj, 
then its limit is a minimum point of F. 


6 Ambrosio and Tortorelli Phase-Field Elliptic 
Approximations 


A specific strategy, closer to the initial formulation of the Mumford-Shah problem in 
terms of pairs (u, K = S,,), is based on the approximation by functionals depending 
on two variables (u, v), the second one related to the set K = S,. 


7 Approximations of the Perimeter by Elliptic Functionals 


The Modica-Mortola theorem [58,59] enables the variational approximation of the 
perimeter functional E +> P(E, Q) = fg |Dyz| < oo of an open subset E of Q by 
the quadratic, elliptic functionals 


W(v) 
Ee: 


mmo) =f (civ + ) as, ve W!7(Q), 


where W(t) is a “double-well” potential. For instance, choosing W(t) = t7(1 — 1), 
assuming that Q is bounded with Lipschitz boundary and setting MM,(v) = oo if 
v € L7(Q) \W!2(Q), the functionals MM,,(v) T’-converge in L7(Q) to 


4 P(E,Q) if v= ye forsome E € B(Q), 
(eve) otherwise, 


F(vy) = 
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where 5(Q2) denotes the o-algebra of Borel subsets of Q. 
Minimizing the functional MM ,(v) with respect to v yields the associated Euler- 
Lagrange equation and boundary condition, 


W'(v) = 2e2Aving, OY, Hon A: 
aN 


which can be easily solved in practice by finite differences. 


8 Ambrosio-Tortorelli Approximations 


In the Mumford and Shah functional, the set K = S, is not necessarily the boundary 
of an open and bounded domain, but a construction similar to MM,(v) can still be 
used, with the potential W(t) = -( —t)? instead. Ambrosio and Tortorelli proposed 
two elliptic approximations [5,6] to the weak formulation of the Mumford and Shah 
problem. Presented is the second one [6], being simpler than the first one [5], and 
commonly used in practice. 

Let ¥ = L?(Q)* and let us define 


Atuv) = [ (u—g)dx +p f [Vad +e f («Iv ? +! a 
Q Q Q de 


(11) 


if (u, v) € W!2(Q)2,0 < v <1, and AT,.(u, v) = + 00 otherwise. 
Also defined is the limiting Mumford-Shah functional, 


F(u,v) = {fo U~ 8x + B fg |Vul’ + oH!"(S,) if uw € SBV(Q), 
+00 otherwise. 
Theorem 4. AT7,I'-converges to F as ¢ ] O in L?(Q). Moreover, AT, admits a 


minimizer (Ue, Vz) such that up to subsequences, ug converges to some u € SBV(Q) 
a minimizer of F(u, 1) and inf AT; (uz, Ve) > F(u, 1). 


Interesting generalizations of this result are given and proved by Braides in [19]. 

In practice, the Euler-Lagrange equations associated with the alternating mini- 
mization of AT, with respect to u = u, and v = v, are used and discretized by finite 
differences. These are 


BATe(o) v) = 2(u = g)- 26 div(v?Vu) =0 


du 


ATU) — 2Bv|Vul? — 2aeAv + £(v— 1). 


One of the finite difference approximations to compute u and v in two dimensions 
X = (X1, X2) is as follows. A time-dependent scheme is used in u = u(x, X2, t) and 
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a stationary semi-implicit fixed-point scheme in v = v(x, x2). Let Ax, = Axy =h 
be the step space, Az be the time step, and g;,;, u/’ j> Yj; be the discrete versions of 
g, and of u and v at iterationn > 0,forl <i < M,1< jj < N. Initialize uo = g 
and v° =0. 
For n => 1, compute and repeat to steady state, fori = 2,..., M —1 andj = 2, 
.., N — 1 (combined with Neumann boundary conditions on dQ): 


A A 2 A 1 2, 
nj2 (“i414 jt 
[Vurli; = ( h a h , 


— n+1 nj2 _ 9ae n n n n 
0 = 2Bvi; |Vu li j 25 Cae PV Tiga Fp psi 


ut aut 5 n B n+1 2 n n n+l 
Be ON SE) ae YE) tag a) 
2 2 


Presented are experimental results obtained using the above Ambrosio-Tortorelli 
approximations applied to the well-known Barbara image shown in Fig. | left. 
Segmented images u are shown in Fig.2, and the corresponding edge sets v are 
shown in Fig.3 for varying coefficients a, B € {1, 5, 10}. We notice that less 
regularization (decreasing both w and 6) gives more edges in v, as expected; thus, 
u is closer to g. Fixed a@ and increasing 6 give smoother image u and fewer edges 
in v. Keeping fixed 6 but varying a does not produce much variation in the results. 
Also shown in Fig. 1 right is the numerical energy versus iterations for the case 
a = B =10, e =0.0001. 

Applications of the Ambrosio-Tortorelli approximations to image restoration will 
be presented in details in Sect. 5. 
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Fig. 1 Left: original image g. Right: numerical energy versus iterations for the Ambrosio- 
Tortorelli approximations (a = B = 10, ¢ = 0.0001) 


9 Level Set Formulations of the Mumford and Shah 
Functional 


In this section are reviewed the level set formulations for minimizing the Mumford 
and Shah functional, as proposed initially by Chan and Vese [29-32,79], and by Tsai 
et al. [78] (see also the related work by Samson et al. [71] and Cohen et al. [36,37]). 
These make the link between curve evolution, active contours, and Mumford-Shah 
segmentation. These models have been proposed by restricting the set of minimizers 
u to specific classes of functions: piecewise constant, piecewise smooth, and with 
the edge set K represented by a union of curves or surfaces that are boundaries of 
open subsets of &. For example, if K is the boundary of an open-bounded subset 
of Q, then it can be represented implicitly, as the zero-level line of a Lipschitz- 
continuous level set function. Thus, the set K as an unknown is substituted by an 
unknown function that defines it implicitly, and the Euler-Lagrange equations with 
respect to the unknowns can be easily computed and discretized. 

Following the level set approach [67, 68, 72,73], let 6: — R be a Lipschitz- 
continuous function. Recalled is the variational level set terminology that will be 
useful to rewrite the Mumford and Shah functional in terms of (u, @), instead of 
(u, K). This is inspired by the work of Zhao et al. [83] on a variational level set 
approach for the motion of triple junctions in the plane. 

Used here is the one-dimensional (1D) Heaviside function H, defined by 


lifz>0 


H = 
=) oi, <0” 


and its distributional derivative 6 = H/ (in the weak sense). In practice, it may be 
necessary to work with smooth approximations of the Heaviside and 6 functions. 
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(1,1) (1,5) (1,10) 
SEGMENTED IMAGE u SEGMENTED IMAGE u SEGMENTED IMAGE u 


, 


(5,1) (5,5) (5,10) 
SEGMENTED IMAGE u SEGMENTED IMAGE u SEGMENTED IMAGE u 


(10,1) (10,5) (10,10) 
SEGMENTED IMAGE u SEGMENTED IMAGE u SEGMENTED IMAGE u 


~. 


Fig. 2. Piecewise-smooth images u as minimizers of the Ambrosio-Tortorelli approximations for 
€ = 0.0001 and various values of (a, 6) 


Used here are the following C™ approximations as ¢ > 0 given by [29,31], 
1 2 
A,(z) = = | 1+ —arctan (=) , O:= Hi, 
2 4 € 


The area (or the volume) of the region {x € Q: d(x) > 0} is 


A{x €Q: o(x) > 0} = [ H@epax. 
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Fig. 3. Corresponding edge sets v as minimizers of the Ambrosio-Tortorelli approximations for 
€ = 0.0001 and various values of (a, B) 


and for a level parameter / € R, the area (or volume) of the region {x € Q: d(x) > 


I} is 


A{x €Q: d(x) > 0} = / A(o(x) — Idx. 
Q 


The perimeter (or more generally the surface area) of the region {x € Q: d(x) > 


0} is given by 
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Lix€ 2: 9x) >0}= f DHE), 


which is the total variation of H(@) in , and the perimeter (or surface area) of 
{x € Q: h(x) > 1} is 


L{x €2:46(x) > R= i, |DH(¢—1)|. 


Given the image data g € L® (Q) C L*(Q) to be segmented, the averages of g over 
the (nonempty) regions {x € Q: d(x) > 0} and {x € Q: h(x) < 0}, respectively, 
are 


Jo 8X) H(G(x)) dx d fo 8) — HG(x)))dx _ fo g(x) H(—$(x)) dx 


JoH@@)dx *  Joa—H@@))dx Jo H(— (x) dx 


More generally, for a given level parameter / € R, the averages of g over the 
corresponding (nonempty) regions {x € Q: d(x) > 1} and {x € Q: d(x) < 1}, 
respectively, are 


Sg Bx) A (h(x) — Idx aol Io g(x) A(l — (x))dx 
Ig H(b(x) — 1) dx Ig Hl — b(x)) dx 


Proved next is that if H and 6 are substituted by the above C® approximations 
H,, 5- as € — 0, approximations of the area and length (perimeter) measures are 
obtained. It is obviously found that H,(z) — A(z) for all z € R, as e — 0, and 
that the approximating area term A,(¢) = Jo H-(b(x))dx converges to A(d) = 
Jo H($(x))dx. 

Generalizing a result of Samson et al. [71], it can be shown [35] that our 
approximating functional L.(@) = fo |DH<()|dx = Jo 5<(@)|Vd|dx converges to 
the length | K| of the zero-level line K = {x € Q: ¢(x) =O}, under the assumption 
that @:Q — R is Lipschitz. The same result holds for the case of any /-level curve 
of ¢@ and not only for the 0-level curve. 


Theorem 5. Let us define 


Oe [ |VH.()| dx = [ 8.($) [Vol dx. 


Then, we have 


tim £.(6) = [ ds = |K|, 
{p=0} 


e—>0 


where K = {x € Q: $(x) = 0}. 
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Proof. Using co-area formula [40], the following is found: 


€ = Fa ds\dp= ie ds \ dp. 
L.(d) [Lf see ‘| ; [pp of ‘ ; 


By setting h(p) = fy—, ds, the following is obtained 


1 ee 
L.(0) = [ sapyh(pydp = [ ship dp. 
R R@e+po- 
By the change of variable 0 = the following is obtained 


lim, +o Le(P) = lime+o fp + qeagrh (Oe) dO = lim,+o fp + apg (Oe) dé 
=h(0) fp +d = h(0)+ arctan 6|$22 = h(0) = So=o ds =|K|, 


which concludes the proof. 


In general, this convergence result is valid for any approximations H,, 6,, under 
the assumptions 


lim He(z) = H(z) inR/{0}, 


8: = Het, Hy € C'(R),{ 25° 6 (x)dx = 1. 

10 Piecewise-Constant Mumford and Shah Segmentation 
Using Level Sets 

Our first formulation is for the case when the unknown set of edges K can be 

represented by K = {x € Q: (x) = 0} for some (unknown) Lipschitz function 

@:Q — R. In this case, the unknown minimizers u to functions are restricted taking 


two unknown values c;, c2, and the corresponding Mumford-Shah minimization 
problem can be expressed as [29, 31] 


inf E(¢1,¢2,¢) = / (g(x) — 1)? H)dx + / (g(x) — 2)" H(-6)dx 
c1,c2. Q Q 


+0 | DH). (12) 
Q 
The unknown minimizer u is expressed as 


u(x) = c, H(p(x)) + col — H(p(x))) = c1 H(p(x)) + oH (—$(x)). 
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#7 is substituted by its C°° approximation H, and instead 


Ee(c1,€2,9) = fo (g(x) — ¢1)? He(p)dx + Jo (g(x) — €2)° He(—b) dx 


(13) 
+Vo de |V H.(p)| dx. 
is minimized. 
The associated Euler-Lagrange equations with respect to c1, C2, and ¢ are 
fo (x) He(b(x))dx fa g(x) He(—G(x))dx 
= A adx = Toad 
and, after simplifications, 
2 2 . Ve : 
6-(¢) (ets) =c) =(2(x) =e)" =div (=) =0inQ, (14) 


with boundary conditions V¢- N =0 on @Q. Since 5, > 0 as defined, the factor 
6-(@) can be removed from (14) or substituted by |V@| to obtain a more geometric 
motion extended to all level lines of ¢, as in the standard level set approach. 

This approach has been generalized by Chung and Vese in [34, 35], where more 
than one-level line of the same level set function ¢@ can be used to represent the edge 
set K. Using m distinct real levels {1; < l, <... < lm}, the function ¢ partitions 
the domain Q into the following m + 1| disjoint open regions, making up Q, together 
with their boundaries: 


Qo = {x € Q: —o0 < P(x) < Ih}, 
Q;, = {x €Q: 1; < o(x) < 1j4i}, 1<j<m-1 
Qm = {x € Q: In < &(x) < +00} 


The energy to minimize in this case, depending on co, C1, ..., Cm, &, Will be 
2 m—1 
E (cori ---6€m) = J g(x) — col? Hh — o@))dx + 18) 
2 
~cj HO) — [Hj boa + f Ia(s) 
Q 


—Cm|? H(b(x) — Im)dx + vo ae |DH(¢ —1))|. 


15 
The segmented image will be given by - 


u(x) = coH (lL, — o(x)) + ae cj H(o(x) —1,)A(j41 — o(x)) 


+ Cm A(o(x) —_ Im). 
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As before, to minimize the above energy, the Heaviside function H by H,, as e > 
0, is approximated and substituted. The Euler-Lagrange equations associated with 
the corresponding minimization 


inf E,(co,C1,.--,Cm.®), (16) 


€05C15-+5Cm 


can be expressed as 


_ fa.) Hellh#tt.x)) dx 
Cop) = Fr adi—oea)dx 


bie Jo 80) He($(4,x) lj) He (lj 41-9 (tx) ) dx 
J ~ fg He(Ot.x)—lj) Hej 41-O(.x) dx” 


— fa 8) Heb Ox)—Im) dx 
Cm(@) fg He(b(t.x)—lm)dx” 


and 
0 = |g — col” 8-(1 — was hg —e,) [6-(j+1 — 6) Ae (o — 1;)—] 


6-(¢ —1;) Hej +i — —|g—¢m|? b-( — Im)+ 
v0 71 [be (p-1; sai (#8) oe 4 go 


where N is the exterior unit normal to the boundary 0Q. 

Given here are the details of the numerical algorithm for solving (17) in two 
dimensions (x, y), using gradient descent, in the case of one function @ with two 
levels 7; =0,/2 = 1 > 0. Leth = Ax = Ay be the space steps, At be the time step, 
and ¢ = h. Let (x;, y;) be the discrete points, for 1 <i, j < M,and gj; > g(x, 
yj), i* o(nAt, x;, yj), with n => 0. Recall the usual finite differences formulas 


A‘ di, = bi41j; —Pij, AX bij = bi — Pi-1,;; 
MX bij; = b1j41— $17, AXbi7 = 61,7 — $i,7-1 


Set n = 0, and start with ¢ ; given (defining the initial set of curves). Then, for each 
n > O until steady state: 


1. Compute averages co(”), c1("), and co(”). 
2. Compute ors derived from the finite differences scheme: 


n+ n n n+l n n+l 
vi, $j a $Ry = 6, (or y vo f Ax $3415 —9i. + AY jj 41791. 
“Ar he = ver, > vor | 


+ |gi.j —0|” —|giy — ex” He (1 42,) | +8 (82; -! 


n n+l ¢" n+l 
vo fax (P87) a ay ( Pi Fs 
mn ae te ~\ lver, 


—|8i,j 9” + |gi,j ~e1|’ H, (¢%, 
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n —¢!". 2 ne —¢". 2 
3. Where vor, = (“4 wu) ae (4, 1) . Let 


Ci = : 


Pg TO : oP ties 
h h 


1 
$7 9-1, Oy 41% 1, 
h h 
C3 = u 
ee 


C= 


Cy = 


4. Letm = 4% (5. ( + 5. ( R -1)) gt Sen PEA, AO). 
The main update equation for ¢ becomes 


OF! = b(t, 1m (Crops + Colas + Olja + Cad) 
+A15, ( ri) (i,j 0? (i = H. ( ij -1)) 
+(gi,j — ¢0)?) + At8_ (oP; —2) (—(ei.5 — 00)" + (gig — en? He (68,))]. 


5. And repeat, until steady state is reached. 


This section is concluded with several experimental results obtained using the 
models presented here that act as denoising, segmentation, and active contours. In 
Fig. 4, an experimental result is shown taken from [31] obtained using the binary 
piecewise-constant model (12); we notice how interior contours can be automati- 
cally detected. In Fig. 5, we show an experimental result using the multilayer model 
(15), with m = 2 and two levels /;, 2, applied to the segmentation of a brain 
image. 

The work in [35,79] also shows how the previous Mumford-Shah level set 
approaches can be extended to piecewise-constant segmentation of images with 
triple junctions, several non-nested regions, or with other complex topologies, by 
using two or more level set functions that form a perfect partition of the domain Q. 


11 Piecewise-Smooth Mumford and Shah Segmentation 
Using Level Sets 


Considered first is the corresponding two-dimensional case under the assumption 
that the edges denoted by K in the image can be represented by one-level set 
function ¢, i.e., K = {x € Q|¢(x) = 0}, and followed are the approaches developed 
in parallel by Chan and Vese [32,79] and by Tsai et al. [78], in order to minimize 
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Fig. 4 Detection of different objects in a noisy image, with various convexities and with an interior 
contour which is automatically detected, using only one initial curve. After a short time, an interior 


contour appears inside the torus, and then it expands. Top: g and the evolving contours. Bottom: 
the piecewise-constant approximations u of g over time, given by u = c; H(¢) + oo(1 — A(¢)) 


the general Mumford and Shah model. As in [79], the link between the unknowns 
u and ¢ can be expressed by introducing two functions u* and u~ (see Fig. 6) such 
that 


+ . 
u(x) = $2) FOC) = 0, 
u(x) if (x) <0. 
Assume that u+ and u~ are H! functions on ¢ > 0 and on ¢ < 0, respectively 
(with Sobolev traces up to all boundary points, i.e., up to the boundary {¢ = 0}). 
The following minimization problem can be written 


where 


Et wb) = W? fou ~ a” H(b)dx + 17 fa lum — 8) (1 - HU) dx 
+ fo|Vut| H())dx + fo |VurP (1 — H())dx + v fo |DH(9)| 


is the Mumford-Shah functional restricted to u(x) = ut (x)H(@(x)) +u™ (x)(1 
- H(¢(x)). Minimizing E(u*, u~, ¢) with respect to u*, u~, and ¢, we obtain 
the following Euler-Lagrange equations (embedded in a time-dependent dynamical 
scheme for @): 


w(ut — g) = Autin{x : o(t,x) > 0 ur 9 on{x: o(t,x) =O} UIQ, 


> on 


(17) 


w(u- — g) = Aw in{x : b(t, x) < 0}, 4 = 0 on{x: (t,x) = 0} UIQ, 
(18) 
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Fig. 5 Segmentation of a brain image using one-level set function with two levels. Parameters: 
1, = 0,1, = 25, At = 0.1, vp = 0.1- 2557, 1,500 iterations 
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Fig. 6 The functions u+, uw 


and the zero-level lines of the U=U- 
level set function ¢ for 
piecewise-smooth image <0 U=U" 
partition >>0 
vo" 
U=U o>0 
U=U> 
o<O0 


cae 


ee ee ee 


Fig. 7 Results on a noisy image, using the level set algorithm for the piecewise-smooth Mumford- 
Shah model with one-level set function. The algorithm performs as active contours, denoising, and 
edge detection 


Fe = 800) [09 (TE) — p2 at — al? — [Dut f+ 02 wal + Iver P, 


ot IV¢ 
(19) 
where 0/dn denotes the partial derivative in the normal direction 7 at the corre- 
sponding boundary. We also associate the boundary condition “e = 0 on 02 to 


Eq. (19). 

Shown in Figs. 7 and 8 are experimental results taken from [79] obtained with 
the piecewise-smooth two-phase model. 

There are cases when the boundaries K of regions forming a partition of the 
image could not be represented by the boundary of an open domain. To overcome 
this, several solutions have been proposed in this framework, and mentioned here 
are two of them: (1) in the work by Tsai et al. [78], the minimization of E(ut, u~, b) 
is repeated inside each of the two regions previously computed, and (2) in the work 
of Chan and Vese [79], two or more level set functions are used. For example, in 
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Na ee » 


Fig. 8 Numerical result using the piecewise-smooth Mumford-Shah level set algorithm with one- 
level set function, on a piecewise-smooth real galaxy image 


two dimensions, the problem can be solved using only two-level set functions, and 
we do not have to know a priori how many gray levels the image has (or how many 
segments). The idea is based on the four-color theorem. Based on this observation, 
we can “color” all the regions in a partition using only four “colors,” such that 
any two adjacent regions have different “colors.” Therefore, using two-level set 
functions, we can identify the four “colors” by the following (disjoint) sets: {@1 > 
0, 2 > O}, {hi < 0, d2 < O}, {hi <0, d2 > O}, {1 < 0, 2 < 0}. The boundaries 
of the regions forming the partition will be given by {¢; = 0} U {¢2 = 0}, and this 
will be the set of curves K. Note that, in this particular multiphase formulation of 
the problem, we do not have the problems of “overlapping” or “vacuum” (i.e., the 
phases are disjoint, and their union is the entire domain &, by definition). 

As before, the link between the function u and the four regions can be made by 
introducing four functions ut +, u*+~, u~t, u~—, which are in fact the restrictions 
of u to each of the four phases, as follows (see Fig. 9): 


++ (x), if d(x) > 0 and ¢2(x) > 0, 
T(x), if di(x) > 0 and d2(x) < 0, 
—H(x), if d(x) < 0 and ¢2(x) > 0, 
~~(x), if di(x) < 0 and ¢2(x) <0. 


Uu 
u(x) = 3" 
Uu 
Uu 


Again, using the Heaviside function, the relation between uw; the four functions 
utt,ut—,u-*, u; and the level set functions ¢, and ¢ can be expressed by 


u=u'* H(o,)H(¢2) + ut H(p1) (1 — A(¢2)) + wt (1 — A(o1)) A (g2) + 
(1 — H(¢;)) — H(¢2)) 
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Fig. 9 The functions u+t, 
ut—,u—+,u——, and the 
zero-level lines of the level 
set functions $1, $2 for 
plecewise-smooth image 
partition 


Introduced here is an energy in level set formulation based on the Mumford-Shah 
functional: 


E(u, $1.2) = 2 fo |ut* — g) H(i) H(gx)dx 
+ fo |Vut*|? H(b1)H(g2) dx 
+2 fo |ut — g|” Hb) — H(g2)) dx 
+ Jo [Vut-? Hip) — H(g>)) ax 
+u? Ja |u-* — g|° 0 — Hd) Aga) ax 
+ fo [Vu-* |’ 1 — H(g1)) H(g2)dx 
+m? fo |u-~ — gl? 1 — H(¢1)) — H(d2)) dx 
+ fo |Vu--/ (1 — H(g1)) — H(g2))dx 
+v fo |DH(g1)| + v fo |DH(92)| 


Note that the expression /, |DH(¢1)| + Jo |DH(¢2)| is not exactly the length term 
of K = {x € Q: ¢i(x) = 0 and ¢2(x) = 0}; it is just an approximation and 
simplification. In practice, satisfactory results using the above formula are obtained, 
and the associated Euler-Lagrange equations are simplified. 

The associated Euler-Lagrange equations are obtained as in the previous cases, 
embedded in a dynamic scheme, assuming (tf, x, y) H>—> @; (t, x, y): minimizing 
the energy with respect to the functions u*t, u*+~, u~*, u~~, we have, for each 
fixed t: 


w2(ut* — g) = Au** in {1 > 0,2 > 0}, 2=* = 0 on {$1 =0, bo = 0}, 


{1 = 0, b2 = O}; 
w(ut — g) = Aut in {$) > 0,¢2 < 0}, au = 0 on {¢; = 0, d2 < 0}, 
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{1 = 0, bo = 0}; 
wut — g) = Au™* in {$1 <0, $2 > 0}, %—* = 0 on {1 = 0, go = 0}, 
{1 <0, bo = 0}; 
1? (u—~ — g) = Au ~ in {1 < 0, $2 < 0}, “— = 0 on {pi = 0, go < 0}, 
{hi < 0, b2 = O} 


The Euler-Lagrange equations evolving ¢; and ¢2, embedded in a dynamic scheme, 
are formally: 


4 = §.(d1) [vv (v5) — 2 |wt+ — g|’ Hd) — |Vut t+ Hs) 
—" Jc r= gl? (1 — H(g2)) ~ [Wu |? (1 — H(a)) + |u-t 
—g|? H(po) + [Vu H (da) + 12 wm — gP (1 HCG) + 
|Vu--|? 1 — H(d2))] = 0, 

#2 = 8.(g2) [vv (qy.) — 2 ut = |? H(di) — Yur ? HO) 

— |e 8| H(d2) + |Vut-|? H(g1)— 22 \u-* — g |? (1 -H(p1))— 

|Vu a! (I~ H(@y) 
=p? |u-~ — gl? (1— H(gy)) + |Vu-? (1 - H(@,))] 


It can be shown, by standard techniques of the calculus of variations on the space 
SBV(QQ) (special functions of bounded variations), and a compactness result due to 
Ambrosio [3], that the proposed minimization problems from this section, in the 
level set formulation, have a minimizer. Finally, because there is no uniqueness 
of minimizers, and because the problems are nonconvex, the numerical results 
may depend on the initial choice of the curves and may compute only a local 
minimum. We think that, using the seed initialization (see [79]), the algorithms 
have the tendency of computing a global minimum, most of the times. Additional 
experimental results are shown in [79]. 


12 Case Examples: Variational Image Restoration with 
Segmentation-Based Regularization 


This section focuses on the challenging task of edge-preserving variational image 
restoration. In this context, restoration is referred to as image deblurring and 
denoising, dealing with Gaussian and impulsive noise models. Terms from the 
Mumford-Shah segmentation functional are used as regularizers, reflecting the 
model of piecewise-constant or piecewise-smooth images. 

In the standard model of degradation, the underlying assumptions are the 
linearity and shift invariance of the blur process and the additivity and normal 
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distribution of the noise. Formally, let & be an open-bounded subset of R”. The 
observed image g: Q > R € L® is given by 


g=hxut+n, (20) 


where g is normalized to the hypercube [0, 1], / is the blur kernel such that h(x) > 
0 and f h(x)dx = 1, u: 2 > R% is the (“ideal”) original image, n ~ N(0, 07) 
stands for a white Gaussian noise, and * denotes the convolution operator. The 
restoration problem is the recovery of the original image u given Eq. (20). Non- 
blind image restoration is the problem whenever the blur kernel is known, while 
blind restoration refers to the case of unknown kernel [49,50]. The recovery process 
in the non-blind case is a typical inverse problem where the image u is the minimizer 
of an objective functional of the form 


F (u) = ®(g—hxu)t+ J (Vu). (21) 


The functional consists of fidelity term and a regularizer. The fidelity term @ forces 
the smoothed image h * u to be close to the observed image g. The commonly used 
model of a white Gaussian noise n ~ N(0, a*) leads by the maximum likelihood 
estimation to the minimization of the L? norm of the noise 


2 
®,2 = |lg—hx ull72(@) : (22) 
However, in the case of impulsive noise, some amount of pixels do not obey the 


Gaussian noise model. Minimization of outlier effects can be accomplished by 
replacing the quadratic form (22) with a robust p-function [44], e.g., 


Op = ||g—A*ullpig)- (23) 
The minimization of (22) or (23) alone with respect to u is an inverse problem which 
is known to be ill-posed: small perturbations in the data g may produce unbounded 


variations in the solution. To alleviate this problem, a regularization term can be 
added. The Tikhonov L? stabilizer [77] 


Ji =| |Vul? dx, 
Q 


leads to over smoothing and loss of important edge information. A better edge 
preservation regularizer, the total variation (TV) term, was introduced by Rudin et 
al. [69,70], where the L* norm was replaced by the L! norm of the image gradients 


Jn = | \Vul dx. 
Q 
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Still, although the total variation regularization outperforms the L* norm, the image 
features — the edges — are not explicitly extracted. The edges are implicitly preserved 
only by the image gradients. 

An alternative regularizer is the one used in the Mumford-Shah functional [61, 
63]. Recall that this is accomplished by searching for a pair (u, K) where K C Q 
denotes the set of discontinuities of u, the unknown image, such that u ¢ H "Q\K ), 
K Cc Q closed in 2, and 


G(u, K) = pf |Vul? dx + aH"! (K) < oo. (24) 
Q/K 


In our study, the regularizer to the restoration problem (21) is given by 
Jms = G (u, K), 


its L! variant [2,74], and elliptic or level set approximations of these, as presented 
next. This enables the explicit extraction and preservation of the image edges in 
the course of the restoration process. We show the advantages of this regularizer in 
several applications and noise models (Gaussian and impulsive). 

As has been mentioned, Ambrosio and Tortorelli [6] introduced an elliptic 
approximation G,(u, v) to G(u, K), as e > OT, that is recalled here, 


— 1) 
G; (u,v) = 6 | v2 |Vul? dx + a f (siv9" te o) dx. (25) 
Q Q de 
Replacing the Mumford-Shah regularization term (24) by G,(u, v) yields the 
proposed restoration model 


2 
Fe (u,v) = b(g—haw +B f PvuPds+a f (sive + 8" 


4e 
(26) 
The functional (26) can also be understood from a generalized robust statistics 
viewpoint. This is beyond the scope of this chapter, and the interested reader can 
find the details in [12]. 

The rest of the chapter considers the non-blind restoration problem presented in 
[13] and its generalizations to several more realistic situations. Consider the problem 
of (semi-) blind deconvolution, the case of impulsive noise, the color restoration 
problem, and the case of space-variant blur. Also consider the problem of restoration 
of piecewise-constant images from noisy-blurry data using the level set form of the 
Mumford-Shah regularizer and image restoration using nonlocal Mumford-Shah- 
Ambrosio-Tortorelli regularizers. 
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13 Non-blind Restoration 


Addressed first is the restoration problem with a known blur kernel /: and additive 
Gaussian noise [10, 13]. In this case, the fidelity term is the L? norm of the 
noise (22), and the regularizer Ys = Gy (u,v) (25). The objective functional is 
therefore 


2 
Fo=5 [ te-hawPavee f AivaParte f (eivoes v Y) 


€ 

(27) 
The functional (27) is strictly convex, bounded from below and coercive with 
respect to the functions u and v if the other one is fixed. Following [33], the 
alternate minimization (AM) approach is applied: in each step of the iterative 
procedure, we minimize with respect to one function and keep the other one fixed. 
The minimization is carried out using the Euler-Lagrange (E-L) equations with 
Neumann boundary conditions where uw is initialized as the blurred image g and 
v is initialized to 1. 


ote Gy Hil i ae Ky 0 (28) 
bv 2e 

SF, ; 

5 = (hx u—g) *h(—x,—-y) — 2BV- (vVu) =0 (29) 
u 


Equation (28) is linear with respect to v and can be easily solved after discretization 
by the minimal residual algorithm [81]. The integrodifferential equation (29) can 
be solved by the conjugate-gradient method [13]. The iterative process is stopped 
whenever some convergence criterion is satisfied (e.g., ||u"t! — u”|| < e|{u"|]). 
Figure 10 demonstrates the outcome of the algorithm. The top-left image is the 
blurred image g. The kernel corresponds to horizontal motion blur. The top-right 
image is the reconstruction obtained using total variation (TV) regularization [70, 
80]. The bottom-left image is the outcome of the MS regularizer, with a known blur 
kernel. The bottom-right image shows the associated edge map v determined by the 
algorithm. Acceptable restoration is obtained with both methods. Nevertheless, the 
MS method yields a sharper result and is almost free of “ghosts” (white replications 
of notes) that can be seen in the top-right image (e.g., between the C notes in the 
right part of the top stave). The algorithm can be also applied to 3D images as shown 
in Fig. 11. In this example, the blur kernel was anisotropic 3D Gaussian kernel. 


14 Semi-blind Restoration 


Blind restoration refers to the case when the blur kernel / is not known in advance. 
In addition to being ill-posed with respect to the image, the blind restoration problem 
is ill-posed in the kernel as well. Blind image restoration with joint recovery of the 
image and the kernel, and regularization of both, was presented by You and Kaveh 


Mumford and Shah Model and Its Applications. . . 1571 


=e — PJsus4 ETsus4 


F7sus4 E7Jsus4 


Fig. 10 The case of a known (nine-pixel horizontal motion) blur kernel. Top left: corrupted image. 
Top-right: restoration using the TV method [70,80]. Bottom left: restoration using the MS method. 
Bottom right: edge map produced by the MS method 


Fig. 11 3D restoration of MR image. Left: blurred (0, = 1.0, 0, = 1.0, 0, = 0.2) image. Middle: 
recovered image. Right: edge map 
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[82], followed by Chan and Wong [33]. Chan and Wong suggested to minimize a 
functional consisting of a fidelity term and total variation (L' norm) regularization 
for both the image and the kernel: 


1 
F (u,h) = 5 |A x u— eli@ + asf |Vul dx + ao f |VA| dx. (30) 


By this approach, the recovered kernel is highly dependent on the image character- 
istics. It allows the distribution of edge directions in the image to have an influence 
on the shape of the recovered kernel which may lead to inaccurate restoration [13]. 
Facing the ill-posedness of blind restoration with a general kernel, two approaches 
can be taken. One is to add relevant data; the other is to constrain the solution. In 
many practical situations, the blurring kernel can be modeled by the physics/optics 
of the imaging device and the setup. The blurring kernel can then be constrained 
and described as a member in a class of parametric functions. The blind restoration 
problem is then reduced to a semi-blind one. Let us consider the case of isotropic 
Gaussian blur parameterized by the width o, 


2 
ee io 
e %?,x ER". 


he (x) = 2no2 


The semi-blind objective functional then takes the form [13] 


1 
F, (v,u,o) = 5 ff in tu By dx + G, ur +7 ff VholPds. (31) 


The last term in Eq. (31) stands for the regularization of the kernel, necessary to 
resolve the fundamental ambiguity in the division of the apparent blur between 
the recovered image and the blur kernel. This means that we prefer to reject 
the hypothesis that the blur originates from u, and assume that it is due to the 
convolution with the blur kernel. From the range of possible kernels, we thus select 
a wide one. This preference is represented by the kernel smoothness term: the 
width of the Gaussian corresponds to its smoothness, measured by the L? norm 
of its gradient. The optimization is carried out by using the alternate minimization 
approach. The recovered image u is initialized with g, the edge indicator function 
v 1s initialized with 1, and o with a small number e which reflects a delta function 
kernel. The Euler-Lagrange equations with respect to v and wu are given by (28) 
and (29), respectively. The parameter o is the solution of 


Fs he 
: =) (hg * u— g) a + ye lvhe? dx =0, (32) 
do Q do do 


which can be calculated by the bisection method. The functional (31) is not 
generally convex. Nevertheless, in practical numerical simulations, the algorithm 
converges to visually appealing restoration results as can be seen in the second row 
of Fig. 12. 
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Fig. 12 Semi-blind restoration. Top row: blurred images. Second row: restoration using the semi- 
blind method. Third row: original images. Bottom row: edge maps produced by the semi-blind 
method 


15 Image Restoration with Impulsive Noise 


Consider an image that has been blurred with a known blur kernel A and con- 
taminated by impulsive noise. Salt-and-pepper noise, for instance, is a common 
model for the effects of bit errors in transmission, malfunctioning pixels, and faulty 
memory locations. Image deblurring algorithms that were designed for Gaussian 
noise produce inadequate results with impulsive noise. 

The left image of Fig. 13 is a blurred image contaminated by salt-and-pepper 
noise, and the right image is the outcome of the total variation restoration method 
[80]. A straightforward sequential approach is to first denoise the image and then 
to deblur it. This two-stage method is however prone to failure, especially at high 
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Fig. 13 Current image deblurring algorithms fail in the presence of salt-and-pepper noise. Left: 
blurred image with salt-and-pepper noise. Right: restoration using the TV method [80] 


Fig. 14 The failure of the two-stage approach to salt-and-pepper noise removal and image 
deblurring. Top-left: blurred image. Top-right: blurred image contaminated by salt-and-pepper 
noise. Bottom-left: the outcome of 3 x 3 median filtering, followed by deblurring. Bottom-right: 
the outcome of 5 x 5 median filtering, followed by deblurring 
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noise density. Image denoising using median-type filtering creates distortion that 
depends on the neighborhood size; this error can be strongly amplified by the 
deblurring process. This is illustrated in Fig. 14. The top-left and top-right images 
are the blurred and blurred-noisy images, respectively. The outcome of 3 x 3 median 
filtering followed by total variation deblurring [80] is shown in the bottom left. At 
this noise level, the 3 x 3 neighborhood size of the median filter is insufficient, 
the noise is not entirely removed, and the residual noise is greatly amplified by the 
deblurring process. If the neighborhood size of the median filter increases to 5 x 5, 
the noise is fully removed, but the distortion leads to inadequate deblurring (bottom 
right). 

In a unified variational framework, the “ideal” image u can be approximated as 
the minimizer of the objective functional [12, 14] 


Fur) = [ V(hxu—g) +ndx + G, (u,v). (33) 
Q 


The quadratic data-fidelity term is now replaced by the modified L' norm [64] 
which is robust to outliers, i.e., to impulse noise. The parameter 7 = | enforces the 
differentiability of (33) with respect to u. Optimization of the functional is carried 
out using the Euler-Lagrange equations subject to Neumann boundary conditions: 


F. = 
Fe = 2pvivul+a(* ) -260.a v=o, (34) 
év 2€ 
5F, heu— 
= 2) *h (—x,—y) — 2BV: (v’Vu) = 0. (35) 


Ou [neue tn 


The alternate minimization technique can be applied here as well since the 
functional (33) is convex, bounded from below and coercive with respect to either 
function u or v if the other one is fixed. Equation (34) is obviously linear with respect 
to v. In contrast, (35) is a nonlinear integrodifferential equation. Linearization of this 
equation is carried out using the fixed-point iteration scheme as in [33, 80]. In this 
method, additional iteration index / serves as intermediate stage calculating w’*!. 
u = u! is set in the denominator, and u = u't! elsewhere, where / is the current 


iteration number. Equation (35) can thus be rewritten as 
Hw a= Ge) PSO Aj ise (36) 


where H. is the linear integrodifferential operator 


I+1 
H (v,u') a es * h(—x,—y) — 2BV- (vVu't!) 
V(t xu! 2) +n 
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and 
& 


(nul —g) +n 


G(u') = *h(-x,—y). (37) 


Note that (36) is now a linear integrodifferential equation in u!*!. 

The discretization of Eqs. (34) and (36) yields two systems of linear algebraic 
equations. These systems are solved in alternation, leading to the following iterative 
algorithm [12]: 

Initialization: u° = g, v? = 1. 


1. Solve for v?*+! 


a a 
(28 [Vu"? + = — 2ae A) yitt = = (38) 


2. Set u"t!° = y" and solve for u"*! (iterating on /) 


H (th, unt) yitii+l _ Gg ert) ; (39) 
3. If (ur! _ yn li. <€ a stop. 


The convergence of the algorithm was proved in [14]. Figure 15 demonstrates the 
performance of the algorithm. The top row shows the blurred images with increasing 
salt-and-pepper noise level. The outcome of the restoration algorithm is shown in 
the bottom row. 

A variant of the Mumford-Shah functional in its -convergence approximation 
was suggested by Shah [74]. In this version, the L? norm of |Vu| in (25) was 
replaced by its L' norm in the first term of G, 


— 1) 
Justv (u,v) =6 | v" |Vul dxtaf (sivs? + v=) Ja 
Q Q 4 


€ 
Alicandro et al. [2] proved the ’-convergence of this functional to 


ut — un 
| | 


Just (a) = Bf WWuldxta f an + |D'ul@), 
Q/K xkil+ |ut—-w| 


where ut and u~ denote the image values on two sides of the edge set K, H' is 
the one-dimensional Hausdorff measure, and D“u is the Cantor part of the measure- 
valued derivative Du. The Mumford-Shah and Shah regularizers are compared in 
Fig. 16. The blurred and noisy images are shown in the left column. The results of 
the restoration using the Mumford-Shah stabilizer (MS) are presented in the middle 
column, and the images recovered using the Shah regularizer (MSTV) are shown in 
the right column. 
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Fig. 15 Top row: the Lena image blurred with a pillbox kernel of radius 3 and contaminated 
by salt-and-pepper noise. The noise density is (/eft to right) 0.01, 0.1, and 0.3. Bottom row: the 
corresponding recovered images 


The recovery using both methods is satisfactory, but it can be clearly seen that 
while the Mumford-Shah restoration performs better in the high-frequency image 
content (see the shades, for instance), the Shah restoration attracts the image toward 
the piecewise constant or cartoon limit which yields images much closer to the 
“ideal.” This can be explained by the fact that the Shah regularizer is more robust to 
image gradients and hence eliminates high-frequency contributions. 

The special case of pure impulse denoising (no blur) is demonstrated in Fig. 17. 
The image on the left shows the outcome of the algorithm of [65] with L! norm for 
both the fidelity and regularization, while the recovery using the L! fidelity and MS 
regularizer is shown on the right. It can be observed that the better robustness of the 
MS regularizer leads to better performance in the presence of salt-and-pepper noise. 


16 Color Image Restoration 


The restoration problem is now extended to vector-valued images [9]. In the case 
of color images, the image intensity is defined as u: Q — [0, 1]*. Here g” denotes 
the observed image at channel v € {r, g, b} such that g’ = h * u® +n”. The 
underlying assumption here is that the blur kernel / is common to all of the channels. 
If the noise is randomly located in a random color channel, the fidelity term can be 
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Fig. 16 Left column: the window image blurred with a pillbox kernel of radius 3 and contaminated 
by salt-and-pepper noise. The noise density is (top to bottom) 0.01 and 0.1. Middle column: 
the corresponding recovered images with Mumford-Shah (MS) regularization. Right column: the 
corresponding recovered images with Shah (MSTV) regularization 


Fig. 17 Pure impulse denoising. Left column: restoration using the L! regularization [65]. Right 
column: restoration using the MS regularizer 


modeled as 
®p = i" eo (h xu’ — g’)’dx 
Q v 


in the case of Gaussian noise, and 


1 =f Vhew ey +ndx, 1&1, oa 
Q 
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in the case of impulsive noise. The TV regularization can be generalized to 


Frv(u) = ful ae. (41) 


where 


|Vul| = ween [Vw Pty, <1. (42) 


The color MS regularizer thus takes the form 


_ 4)2 
Ims(u, v) = p | v? Vull” dx+af (civor + G ) dx. (43) 
Q Q 


Note that in this regularizer the edge map v is common for the three channels 
and provides the necessary coupling between colors. In the same fashion, the color 
MSTYV regularizer is given by 


_ 42 
Iusrv (u,v) = p | v? ||Vul dx+af («ivor + o) dx. (44) 
Q Q 


Once again, the optimization technique is alternate minimization with respect to 
u” and v [9]. Figure 18 demonstrates the outcome of the different regularizers for 
an image blurred by Gaussian kernel and corrupted by both Gaussian and salt-and- 
pepper noise. The fidelity term in all cases was selected as ©) (40). 

The methods based on Mumford-Shah regularizer are superior to the TV 
stabilizers, where MSTV provides a result slightly closer to the “ideal” with little 
loss of details. 


17 Space-Variant Restoration 


The assumption of space-invariant blur kernel is sometimes inaccurate in real pho- 
tographic images. For example, when multiple objects move at different velocities 
and in different directions in a scene, one gets space-variant motion blur. Likewise, 
when a camera lens is focused on one specific object, other objects nearer or farther 
away from the lens are not as sharp. In such situations, different blur kernels degrade 
different areas of the image. In some cases, it can be assumed that the blur kernel is a 
piecewise space-variant function. This means that every sub-domain in the image is 
blurred by a different kernel. In the full blind restoration, several operations have to 
be simultaneously applied: (1) segmentation of the subregions, (2) estimation of the 
blur kernels, and (3) recovery of the “ideal” image. Presented here is the simplest 
case where it is assumed that the subregions and blur kernels are known in advance. 
The segmentation procedure in a semi-blind restoration problem can be found in 
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Blurred and noisy 


MS MSTV 


Fig. 18 Recovery of the Lena image blurred by 7 x7 out-of-focus kernel contaminated by mixture 
of Gaussian and salt-and-pepper noise 


[15]. The non-blind space-variant restoration approach relies on the use of a global 
regularizer, which eliminates the requirement of dealing with region boundaries. As 
a result, the continuity of the gray levels in the recovered image is inherent. This 
method does not limit the number of subregions, their geometrical shape, and the 
kernel support size. 

Let the open nonoverlapping subsets w; C Q denote regions that are blurred 
by kernels h;, respectively. In addition, Q/Uw; denotes the background region 
blurred by the background kernel hy, and w; stands for the closure of w;. The 
region boundaries are denoted by dw;. The recovered image u is the minimizer of 
the objective functional 


I 2 Nb 2 
Fl(u,v) = ap a Ni i. (A; *u— g)dx + 5 Doom (h, xu— g)dx+Ty. 
(45) 
where 7; and np are positive scalars and ys(u,v) is the Mumford-Shah regular- 
izer (25). Following the formulation of Chan and Vese [31], the domains w; can be 
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Fig. 19 Non-blind space-variant restoration. Left column: spatially variant motion blurred images. 
Right column: the corresponding recovered images using the suggested method 


replaced by the Heaviside function H(¢;), where 


1, d > 0, 


A(i) = 0, #: <0, (46) 
and ¢; : Q — Risa level set function such that 
da; = {x € Q: g(x) = O}. 
The functional then takes the form 
Fu,v) = 5 Di mi fo (hi xu gPH (pi)dx+ as 


mh fo (hp xu—g)(1— do; H(g;))dx + Ius(u, v). 


Figure 19 demonstrates the performance of the suggested algorithm. The two 
images in the left column were synthetically blurred by different blur kernels within 
the marked shapes. The corresponding recovered images are shown in the right 
column. Special handling of the region boundaries was not necessary because the 
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MS regularizer was applied globally to the whole image, enforcing the piecewise- 
smoothness constraint. This means that the boundaries of the blurred regions were 
smoothed within the restoration process while edges were preserved. 


18 Level Set Formulations for Joint Restoration and 
Segmentation 


Presented here are other joint formulations for denoising, deblurring, and piecewise- 
constant segmentation introduced in [45] that can be seen as applications and 
modifications of the piecewise-constant Mumford-Shah model in level set for- 
mulation presented in Sect. 10. For related discussion, the reader is referred to 
[11-13, 47,53]. A minimization approach is used, and the gradient descent method 
is considered. Let g = h * u +n be a given blurred-noisy image, where h is a 
known blurring kernel (such as the Gaussian kernel) and n represents Gaussian 
additive noise of zero mean. We assume that the contours or jumps in the image 
u can be represented by the m distinct levels {- co = Ib < 1, < bh < 

< In < In+1 = 00} of the same implicit (Lipschitz-continuous) function 
g:Q — R partitioning Q into m + 1 disjoint open regions R; = {x € Q: 
[j-1 < o(x) < 1j}, 1 < j < m+ 1. Thus, the denoised-deblurred image 
u = (H(g - Im) + wij A(d — Im—j+1) A (lm—j+2 —@)+ Cm — p) 
is recovered by minimizing the following energy functional (vp > 0): 


E(c\,¢2,... ,Cm+1, 9) = Te lg—h * (a1 H(¢d-— In) + Vici A = lm=j+ 
t+em+1 A (hi — $))Pdx + vo 4 So IVA (b — 1)) 


In the binary case (one level m = 1, /; = 0), we assume the degradation model g = h 
*c,H(o)+c2.(1— H(¢)) + n, and we wish to recover u = c; H(@) +c2(1— H(¢)) 
in Q together with a segmentation of g. The modified binary segmentation model 
incorporating the blur becomes 


info, .o.6{E (C1, 02,0) = fo lg —h * (c1H(¢) + (1 — H(g))) |? dx 


(48) 
+0 fa |VH(6)|dz}. 


The Euler-Lagrange equations are computed minimizing this energy with respect to 
C1, C2, and @. Using alternating minimization, keeping first ¢ fixed and minimizing 
the energy with respect to the unknown constants c; and c2, the following linear 
system of equations are obtained: 


C1 fg h{dx + co fo hihgdx = f ghidx, 
C1 fg hihodx + 2 fg h3dx = f ghodx 


with the notations h; = h * H(@) and hy = h * (1 — H(@)). Note that the linear 
system has a unique solution because the determinant of the coefficient matrix is not 
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Fig. 20 Joint segmentation, I r t - = 
denoising, and deblurring in Letters in 
wsingthebinarylevelset’ = Mathematical Mathematical 


model. Top row: (from left to 


right) degraded image g Physics Physics 


(blurred with motion blur 

kernel of length 10, oriented 

at an angle 9 = 25° wrt. the 

horizon and contaminated by 

Gaussian noise with (Physies ) eS 
oO, = 10), original image. 

Rows 2-5: initial curves, 


curve evolution using (48) at 
iterations 50, 100, and 300 


with vp = 5-255? and the (Ph 

restored image 

u(SNR = 28.1827). (¢1,¢2); Letters in “ \ 
original image ~ (62.7525, 

259.8939), restored u, ‘Mathematical 
(61.9194, 262.7795) Physies 


Physics Physics 


zero due to the Cauchy-Schwartz inequality fo hyh2 dx? < fo hi dxfo h3 dx, where 
the equality holds if and only if hy = ho for ae. x € Q. But clearly, hy = h * 
A(#)) and hy = h * (1 — H(@)) are distinct; thus, we have strict inequality. 

Keeping now the constants c; and cz fixed and minimizing the energy with 
respect to @, the evolution equation is obtained by introducing an artificial time 
for the gradient descent in @(t, x),t > 0,x €Q 


P(t, x) = 6(6) | (i * g — crh * (h* H(g)) — coh « (h (1— H@)))) 
(ci — 2) + vodiv (y4r) |. 


where h(x) = h(—x). 

Figure 20 shows a numerical result for joint denoising, deblurring, and segmen- 
tation of a synthetic image, in a binary level set approach. 

In the case of two distinct levels /; < /, of the level set function ¢(m = 2), we 
wish to recover a piecewise-constant image of the form u = c; H(¢@—lh) +o. H(h— 
$)H(¢ -1;)+c3H(1,—¢) and a segmentation of g, assuming the degradation model 
g=h*yA(o—-h)+oH(bh-@A@-1)+cAH(l,—) +n, by minimizing 


infer c,c3,¢ E (C1, €2,¢3,0) = fo lg —h * (A A (@ — hh) +0. (bh — ¢) H (ohh) 
+¢3H (I) — ))/’ dx + vo Saeen So lVH (¢ -1;)| 4x. 
(49 
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Fig. 21 Original image (/eft) and its noisy, blurry version (right) blurred with Gaussian kernel 
with o, = | and contaminated by Gaussian noise o,, = 20 


Similar to the previous binary model with blur, for fixed ¢, the unknown constants 
are computed by solving the linear system of three equations: 


cy f hidx + co f hihodx + c3 f hih3dx = f ghidx 
cy f hyhodx + co f hjdx + c3 f hoh3dx = f ghodx 
cy f hyh3dx + C2 f hoh3dx + c3 f h3dx = [ gh3dx 


where hy = h* H(b—1y), hp =h* H(l,—)H(h—|,), andh3 = h* H(l,-@). 
For fixed cj, cz, and c3, by minimizing the functional E with respect to @, the 
gradient descent is obtained for @(t, x), tf > 0,x € Q: 


% (t,x) =h* (g—h* (cH (¢ —b) + OH (b- 6) H @—-b) 
+c3H (I) — $)) (C18 ($— by) 
+e. (1, — 6)8 (6 —h)) — 2H (¢ — 1) 6 (hh — 6) — 36 (h — #))) 
+vo div (1975;) (8(@—h) +8 (@—h)). 


(50) 


Figures 21 and 22 show a numerical result for joint denoising, deblurring, and 
segmentation of the brain image in a multilayer level set approach. 


19 Image Restoration by Nonlocal Mumford-Shah 
Regularizers 


The traditional regularization terms discussed in the previous sections (depending 
on the image gradient) are based on local image operators, which denoise and 
preserve edges very well, but may induce loss of fine structures like texture 
during the restoration process. Recently, Buades et al. [22] introduced the nonlocal 
means filter, which produces excellent denoising results. Gilboa and Osher [42, 43] 
formulated the variational framework of NL means by proposing nonlocal regu- 
larizing functionals and the nonlocal operators such as the nonlocal gradient and 
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Fig. 22 Curve evolution and 
restored u using (51), 

Vo = 0.022557, (c1, C2, €3): 
original image ~ 

(12.7501, 125.3610, 255.6453), 
restored u ~ 

(22.4797, 136.9884, 255.0074) 


divergence. Following Jung et al. [46], presented here are nonlocal versions of the 
Mumford-Shah-Ambrosio-Tortorelli regularizing functionals, called NL/MSH! and 
NL/MSTV, by applying the nonlocal operators proposed by Gilboa-Osher to MSH! 
and MSTYV, respectively, for image restoration in the presence of blur and Gaussian 
or impulse noise. In addition, for the impulse noise model, use of a preprocessed 
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image is proposed to compute the weights w (the weights w defined in the NL means 
filter are more appropriate for the additive Gaussian noise case). 
First recall the Ambrosio-Tortorelli regularizer, 


—1) 
MSH (yy) = pf VP Wiledx ta f (sive + oi) dx, (51) 
Q Q 


where 0 < v(x) < | represents the edges: v(x) = Oif x € K and v(x) & 1 
otherwise, € is a small positive constant, and a, 6 are positive weights. 

Shah [74] suggested a modified version of the approximation (51) to the MS 
functional by replacing the norm square of |Vu| by the norm in the first term: 


2 
WMSTV (14, vy) =6 | v |Vul dx+a f (sivoP + @7!) Jaw 
Q Q 


de 


This functional I converges to the other MSY functional [2]: 


ut —un 
| | 


WS) = pf (Walder f ATA dit! + |Deul (2). 


where u* and u~ denote the image values on two sides of the jump set K = J, of 
uand D_u is the Cantor part of the measure-valued derivative Du. 

Nonlocal methods in image processing have been explored in many papers 
because they are well adapted to texture denoising, while the standard denoising 
models working with local image information seem to consider texture as noise, 
which results in losing texture. Nonlocal methods are generalized from the neigh- 
borhood filters and patch-based methods. The idea of neighborhood filter is to 
restore a pixel by averaging the values of neighboring pixels with a similar gray 
level value. 

Buades et al. [22] generalized this idea by applying the patch-based methods, 
proposing a famous neighborhood filter called nonlocal means (or NL means): 


_ da (ulx).u(y)) 


NlLux)=calee " ul)dy 
da (u(x) ,u(y)) = fer Ga (t) u(x +1) u(y +0)? dt 


where d, is the patch distance, G, is the Gaussian kernel with standard deviation a 
a da (u(x),u(y)) . . . 
determining the patch size, C (x) = f[, Qe iz dy is the normalization factor, 


and h is the filtering parameter which corresponds to the noise level; it is usually set 
to be the standard deviation of the noise. The NL means not only compares the gray 
level at a single point but the geometrical configuration in a whole neighborhood 
(patch). Thus, to denoise a pixel, it is better to average the nearby pixels with similar 
structures rather than just with similar intensities. 
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In practice, the search window Q, = {y € Q: |y — x| < r} is used instead of 
Q (semi-local) and the weight function at (x, y) € Q x Q depending on a function 
u:Q > R 


7 da (u(x) u(y) 
w(x, y) = exp a 


The weight function w(x, y) gives the similarity of image features between two 
pixels x and y, which is normally computed based on the blurry-noisy image g. 

Based on the gradient and divergence definitions on graphs in the context 
of machine learning, Gilboa and Osher [43] derived the nonlocal operators. Let 
u:Q > R be a function, and w:Q x Q — R is a weight function assumed to be 
nonnegative and symmetric. The nonlocal gradient V,u:Q x Q — R is defined 
as the vector (V,,u) (x, y) := (u(y) —u(x)) /w (x, y). Hence, the norm of the 
nonlocal gradient of u at x € Q is defined as 


[Vil (x) = if (u(y) — u(x)” w(x, y) dy. 


The nonlocal divergence div,,v:2 — R of the vector v:2 x Q — R is defined as 
the adjoint of the nonlocal gradient 


(div,3) (2) = [ (x,y) v0.2) fw dy. 


Based on these nonlocal operators, they introduced nonlocal regularizing function- 
als of the general form 


W(u) = [ (ivou) dx, 


where (s) is a positive function, convex in ./s with @(0) = 0. Inspired by these 
ideas, nonlocal versions of Ambrosio-Tortorelli and Shah approximations to the MS 
regularizer for image denoising-deblurring are presented. This is also continuation 
of work by Bar et al. [11-13], as presented in the first part of this section. 

Proposed are the following nonlocal approximated Mumford-Shah and 
Ambrosio-Tortorelli regularizing functionals (NL/MS) by applying the nonlocal 
operators to the approximations of the MS regularizer, 


2 
WNL/MS (yy) = p | vo (|Vul*) dx + af (: |VvP> + oo) dx, 
Q Q de 
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where $(s) = s and ¢ (s) = ./s correspond to the nonlocal version of MSH! and 
MSTV regularizers, called here NL/MSH! and NL/MSTY, respectively: 


2 
NL/MSH! (yyy) = pf Vv? |Vypul? dx + af (siv? + Oe las 
Q Q 4e 


(v—1) 


WQNL/MSTV (4) y) = 6 | v? |Vipul dx +af (sivo? + Oo lan. 
Q Q de 


In addition, these nonlocal regularizers are used to deblur images in the presence 
of Gaussian or impulse noise. Thus, by incorporating the proper fidelity term 
depending on the noise model, two types of total energies as 

Gaussian noise model: 


E® (u,v) = i (g —h x u)dx + VMS (yy), 
Q 
Impulse noise model: 
E™ (u,v) = i lg —h «ul dx + WMS (u,v). 
Q 


Minimizing these functionals in u and v, the Euler-Lagrange equations: 
Gaussian noise model: 


dE° 2 y-1 

ao = 2b re (IVuul?) — 20a Av +o (54) = 0, 
G 

~ = h* *(h*xu—g) + LNUMS, = 0 
u 


Impulse noise model: 


oE™ 5 
Ay = 2Bvd (1Vive ) -2e0 Av +a (4) = 0, 
aEm™ 
F = h* x sign (h *U —g) en LNUMS,, = 0, 
Uu 


where h* (x) = h(—x) and 


LNUMS,, — 2 Ig Uy) — u(x) w(x, y) 
[(? 0) 4" (1% CP) 
+1 (x) $! (|Vn wl? @)) Jay 
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More specifically, the NL/MSH! and NL/MSTV regularizers give 


LNL/MSH! _ —2Vy . (? (x) Vit (x)) 
= —2 f, (u(y) — u(x))w (x, y) 
[v? () + v? (x) dy, 


Vw 
LMIMSTYn = —Vy- (v2 (x) EE) 


== fo wv) — ux) wee) [ bes + wet | ay 


The energy functionals © (u, v) and E!™ (u, v) are convex in each variable and 
bounded from below. Therefore, to solve two Euler-Lagrange equations simulta- 
neously, the alternate minimization (AM) approach is applied: in each step of the 
iterative procedure, we minimize with respect to one function while keeping the 
other one fixed. Due to its simplicity, the explicit scheme for u based on the gradient 
descent method and the Gauss-Seidel scheme for v is used. Note that since both 
energy functionals are not convex in the joint variable, only a local minimizer may 
be computed. However, this is not a drawback in practice, since the initial guess for 
u in our algorithm is the data g. 

Furthermore, to extend the nonlocal methods to the impulse noise case, a 
preprocessing step is needed for the weight function w(x, y) since it is not possible 
to directly use the data g to compute w. In other words, in the presence of impulse 
noise, the noisy pixels tend to have larger weights than the other neighboring points, 
so it is likely to keep the noise value at such pixel. Thus, a simple algorithm is 
proposed to obtain first a preprocessed image f, which removes the impulse noise 
(outliers) as well as preserves the textures as much as possible. Basically, the median 
filter is used, well known for removing impulse noise. However, if one step of the 
median filter is applied, then the output may be too smoothed out. In order to pre- 
serve the fine structures as well as to remove the noise properly, the idea of Bregman 


Fig. 23 Original and noisy-blurry images (noisy-blurry image using the pillbox kernel of radius 
2 and Gaussian noise with o, = 5) 
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Fig. 24 Recovery of noisy-blurry image from Figs.21-23. Top row: recovered image u using 
MSTV (SNR = 25.1968), MSH!(SNR = 23.1324). Third row: recovered image u using NL/MSTV 
(SNR = 26.4696), NL/MSH!(SNR = 24.7164). Second, bottom rows: corresponding residuals g—h 
* 4. B =0.0045 (MSTV), 0.001 (NL/MSTV), 0.06 (MSH!), 0.006 (NL/MSH!), a =0.00000001, 
€=0.00002 
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Fig. 25 Recovery of noisy-blurry image with Gaussian kernel with o = 1 and salt-and-pepper 
noise with d = 0.3. Top row: original image, blurry image, noisy-blurry image. Middle row: 
recovered images using MSTV (SNR = 27.8336), MSH!(SNR = 23.2052). Bottom row: recovered 
images using NL/MSTV (SNR=29.3503), NL/MSH!(SNR = 27.1477). Parameters: B=0.25 
(MSTV), 0.1 (NL/MSTV), a = 0.01, ¢ = 0.002. Parameters: B = 2 (MSH'), 0.55 (NL/MSH!), 
a = 0.001, e = 0.0001 


iteration is used [21,66], and the following algorithm is proposed to obtain a prepro- 
cessed image f that will be used only in the computation of the weight function: 


¢ Initialize: ro =0, fo =0. 

¢ do (iterate n =0, 1,2,...) 

© fnt1 = median(g + rn, [aa]) 

© m4i=Mmtg-h* fori 

¢ while ||g—hs fnlli > |[@ — A« frill 
¢ Optional f,, = median(f,, [bb]) 


where g is the given noisy-blurry data and median (u, [aa]) is the median filter of 
size a X a with input u; the optional step is needed in the case when the final f,,, still 
has some salt-and-pepper-like noise. This algorithm is simple and requires a few 
iterations only, so it takes less than 1 s for a 256 x 256 size image. The preprocessed 
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\ | \ 


Fig. 26 Edge map v using the MS regularizers in the recovery of the Lena image blurred with 
Gaussian blur kernel with o, = 1 and contaminated by salt-and-pepper noise with density d = 
0.3. Top: (left) MSTV, (right) NL/MSTV. Bottom: (left) MSH!, (right) NL/MSH! 


image f will be used only in the computation of the weights w while keeping g in 
the data-fidelity term; thus, artifacts are not introduced by the median filter. 

Figures 23 and 24 show an experimental result for image restoration of a boat 
image degraded by the pillbox kernel blur of radius 2 and additive Gaussian noise. 
The nonlocal methods give better reconstruction. 

Figures 25 and 26 show an experimental result for image restoration of a woman 
image degraded by Gaussian kernel blur and salt-and-pepper noise. Figure 26 shows 
the edge set v for the four results. The nonlocal methods give better reconstruction. 

Figure 27 shows an experimental result for restoration of the Einstein image 
degraded by motion kernel blur and random-valued impulse noise. The nonlocal 
methods give better reconstruction. 


20 Conclusion 


This chapter is concluded by first summarizing its main results. The Mumford-Shah 
model for image segmentation has been presented, together with its main properties. 
Several approximations to the Mumford and Shah energy have been discussed, with 
an emphasis on phase-field approximations and level set approximations. Several 
numerical results for image segmentation by these methods have been presented. 
In the last section of the chapter, several restoration problems were addressed in 
a variational framework. The fidelity term was formulated according to the noise 
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Fig. 27 Comparison between MSH! and NL/MSH! with the image blurred and contaminated by 
high density (d = 0.4) of random-valued impulse noise. Top: noisy-blurry image blurred with 
the motion blur in recovered images using MSH! (left, SNR = 17.9608) and NL/MSH! (right, 
SNR = 20.7563). Bottom: noisy blurry image blurred with the Gaussian blur in recovered images 
using MSH! (left, SNR = 16.6960) and NL/MSH! (right, SNR = 24.2500). Top: B = 1.5 (MSH!), 
0.5 (NL/MSH!), a = 0.0001, ¢ = 0.002. Bottom: B = 2.5 (MSH!'), 0.65 (NL/MSH!), a = 
0.000001, ¢ = 0.002 


model (Gaussian, impulse, multichannel impulse). First, the a priori piecewise- 
smooth image model was mathematically integrated into the functional as an 
approximation of the Mumford-Shah segmentation elements by the ’-convergence 
formulation. Comparative experimental results show the superiority of this reg- 
ularizer with respect to modern state-of-the-art restoration techniques. Also, the 
piecewise-constant level set formulations of the Mumford-Shah energy have been 
applied to image restoration (related to relevant work by Kim et al. [47]), joint with 
segmentation. Finally, in the last section, the Ambrosio-Tortorelli approximations 
and Bar et al. restoration models have been extended to nonlocal regularizers, 
inspired by the work of Gilboa et al. These models produce much improved 
restoration results for images with texture and fine details. 


21 Recommended Reading 


Many more topics on the Mumford-Shah model and its applications have been 
explored in image processing, computer vision, and more generally inverse prob- 
lems. This chapter contains only a small sample of results and methods. As 
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m 
th 


entioned before, detailed monographs on the Mumford-Shah problem and related 
eoretical and application topics by Blake and Zisserman [16], by Morel and 


Solimini [60], by Chambolle [26], by Ambrosio et al. [4], by David [38], and by 
Braides [19] are recommended. Also, the monographs by Aubert and Kornprobst [8] 
and by Chan and Shen [28] contain chapters presenting the Mumford and Shah 
problem and its main properties. 


The authors would like to mention the work by Cohen et al. [36, 37] on 


using curve evolution approach and the Mumford-Shah functional for detecting 


th 


e boundary of a lake. The work by Aubert et al. [7] also proposes an interesting 


approximation of the Mumford-Shah energy by a family of discrete edge-preserving 
functionals, with '-convergence result. 
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Abstract 

Denoising images can be achieved by a spatial averaging of nearby pixels. How- 
ever, although this method removes noise, it creates blur. Hence, neighborhood 
filters are usually preferred. These filters perform an average of neighboring 
pixels, but only under the condition that their gray level is close enough to the one 
of the pixel in restoration. This very popular method unfortunately creates shocks 
and staircasing effects. It also excessivelly blurs texture and fine structures when 
noise dominates the signal. 

In this chapter, we perform an asymptotic analysis of neighborhood filters 
as the size of the neighborhood shrinks to zero. We prove that these filters are 
asymptotically equivalent to the Perona-Malik equation, one of the first nonlinear 
PDEs proposed for image restoration. As a solution to the shock effect, we 
propose an extremely simple variant of the neighborhood filter using a linear 
regression instead of an average. By analyzing its subjacent PDE, we prove that 
this variant does not create shocks: it is actually related to the mean curvature 
motion. 

We also present a generalization of neighborhood filters, the nonlocal means 
(NL-means) algorithm, addressing the preservation of structure in a digital 
image. The NL-means algorithm tries to take advantage of the high degree of 
redundancy of any natural image. By this, we simply mean that every small 
window in a natural image has many similar windows in the same image. Now 
in a very general sense inspired by the neighborhood filters, one can define as 
“neighborhood of a pixel” any set of pixels with a similar window around. All 
pixels in that neighborhood can be used for predicting its denoised value. 

We finally analyze the recently introduced variational formulations of neigh- 
borhood filters and their application to segmentation and seed diffusion. 


1 Introduction 


The neighborhood filter or sigma filter is attributed to J.S. Lee [48] (in 1983) but 
goes back to L. Yaroslavsky and the Sovietic image-processing theory [75]. This 
filter is introduced in a denoising framework for the removal of additive white noise: 


v(x) = u(x) + n(x), 


where x indicates a pixel site, v(x) is the noisy value, u(x) is the “true” value at 
pixel x, and n(x) is the noise perturbation. When the noise values n(x) and n(y) at 
different pixels are assumed to be independent random variables and independent 
of the image value u(x), one talks about “white noise.” Generally, n(x) is supposed 
to follow a Gaussian distribution of zero mean and standard deviation o. 

Lee and Yaroslavsky proposed to smooth the noisy image by averaging only 
those neighboring pixels that have a similar intensity. Averaging is the principle of 
most denoising methods. The variance law in probability theory ensures that if N 
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Fig. 1 The nine pixels in the baboon image on the right have been enlarged. They present a high 
red-blue contrast. In the red pixels, the first (red) component is stronger. In the blue pixels, the third 
component, blue, dominates 


noise values are averaged, the noise standard deviation is divided by VN. Thus, one 
should, for example, find for each pixel nine other pixels in the image with the same 
color (up to the fluctuations due to noise) in order to reduce the noise by a factor 
3. A first idea might be to chose the closest ones. Now, the closest pixels have not 
necessarily the same color as illustrated in Fig. 1. Look at the red pixel placed in the 
middle of Fig. 1. This pixel has five red neighbors and three blue ones. If the color 
of this pixel is replaced by the average of the colors of its neighbors, it turns blue. 
The same process would likewise redden the blue pixels of this figure. Thus, the 
red and blue border would be blurred. It is clear that in order to denoise the central 
red pixel, it is better to average the color of this pixel with the nearby red pixels 
and only them, excluding the blue ones. This is exactly the technique proposed by 
neighborhood filters. 

The original sigma and neighborhood filter were proposed as an average of the 
spatially close pixels with a gray level difference lower than a certain threshold h. 
Thus, for a certain pixel x, the denoised value is the average of pixels in the spatial 
and intensity neighborhood: 


ty € Q| |x — yl] < p and|u(x) — u(y)| < A}. 


However, in order to make it coherent with further extensions and facilitate the 
mathematical development of this chapter, we will write the filter in a continuous 
framework under a weighted average form. We will denote the neighborhood or 
sigma filter by NF and define it for a pixel x as 


[wou 


1 
NE gt) = Boy fp move m dy, (1) 
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where B,(x) is a ball of center x and radius p > 0, > 0 is the filtering parameter, 
|u(y) =u]? 


and C@) = f, By(x) © ~~~ dy is the normalization factor. The parameter h 
controls the degree of color similarity needed to be taken into account in the average. 
This value depends on the noise standard deviation o, and it was set to 2.50 in [48] 
and [75]. 

The Yaroslavsky and Lee’s filter (1) is less known than more recent versions, 
namely, the SUSAN filter [68] and the bilateral filter [70]. Both algorithms, instead 
of considering a fixed spatial neighborhood B,(x), weigh the distance to the 
reference pixel x: 


yo? Leoy—ueo|? 
i aaa ce 4 
y, 


1 = > 
SFp pu(x) = ae fore roe (2) 


yTx 2 u(y)—u(x : 
where C(x) = fo Pa a Pat ee dy is the normalization factor and p > 0 
is now a spatial filtering parameter. Even if the SUSAN algorithm was previously 
introduced, the whole literature refers to it as the bilateral filter. Therefore, we shall 
call this filter by the latter name in subsequent sections. 

The only difference between the neighborhood filter and the bilateral or SUSAN 
filter is the way the spatial component is treated. While for the neighborhood filter 
all pixels within a certain spatial distance are treated uniformly, for the bilateral or 
SUSAN filter, pixels closer to the reference one are considered more important. We 
display in Fig. 2 a denoising experience where a Gaussian white noise of standard 
deviation 10 has been added to a non-noisy image. We display the denoised image 
by both the neighborhood and bilateral filters. We observe that both filters avoid 
the excessive blurring caused by a Gaussian convolution and preserve all contrasted 
edges in the image. 


Fig. 2 From left to right: noise image, Gaussian convolution, neighborhood filter, and bilateral 
filter. The neighborhood and bilateral filters avoid the excessive blurring caused by a Gaussian 
convolution and preserve all contrasted edges in the image 
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The above denoising experience was applied to color images. In order to clarify 
how the neighborhood filters are implemented in this case, we remind that each pixel 
x is a triplet of values u(x) = (u; (Xx), u2(x), u3(X)), denoting the red, green, and blue 
components. Then, the filter rewrites 


Jeo) wx) |]? 
ne d 


uj (y)e y, 


1 
NF»), pti (x) = C(x) Box) 
mes 


||u(y) — u(x)||* being the average of the distances of the three channels: 


u(y) — u(xpl? = 5 bei) = 00) 


The same definition applies for the SUSAN or bilateral filter by incorporating 
the spatial weighting term. The above definition naturally extends to multispectral 
images with an arbitrary number of channels. Bennett et al. [7] applied it to 
multispectral data with an infrared channel and Peng et al. [56] for general 
multispectral data. 

The evaluation of the denoising performance of neighborhood filters and compar- 
ison with state-of-the-art algorithms are postponed to Sect. 2. In the same section, 
we present a natural extension of the neighborhood filter, the NL-means algorithm, 
proposed in [12]. This algorithm evaluates the similarity between two pixels x and 
y not only by the intensity or color difference of x and y but by the difference of 
intensities in a whole spatial neighborhood. 

The bilateral filter was also proposed as a filtering algorithm with a filtering 
scale depending on both parameters h and p. Thus, taking several values for these 
parameters, we obtain different filtered images and corresponding residuals in a 
multi-scale framework. In Fig.3, we display several applications of the bilateral 
filter for different values of the parameters / and p. We also display the differences 
between the original and filtered images in Fig.4. For moderated values of h, this 
residual contains details and texture, but it does not contain contrasted edges. This 
contrasted information is removed by the bilateral filter only for large values of h. 
In that case, all pixels are judged as having a similar intensity level and the weight is 
set taking into account only the spatial component. It is well known that the residual 
by such an average is proportional to the Laplacian of the image. In Sect. 2, we will 
mathematically analyze the asymptotical expansion of the neighborhood residual 
image. 

This detail removal of the bilateral while conserving very contrasted edges is 
the key in many image and video processing algorithms. Durand and Dorsey [28] 
use this property in the context of tone mapping whose goal is to compress the 
intensity values of a high-dynamic-range image. The authors isolate the details 
before compressing the range of the image. Filtered details and texture are added 
back at the final stage. Similar approaches for image editing are presented by Bae 
et al. [5], which transfer the visual look of an artist picture onto a casual photograph. 
Eisemann and Durand [32] and Petschnigg et al. [58] combine the filtered and 
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Qe li 


Fig. 3 Several applications of the bilateral filter for increasing values of parameters p and h. The 
parameter p increases from top to bottom taking values {2, 5, 10} and A increases from left to right 
taking values {5, 10, 25, 100} 


oe ie 


residual image of a flash and non-flash image of the same scene. These two last 
algorithms, in addition, compute the weight configuration in one image of the 
pair and average the intensity values of the other image. As we will see, this is a 
common feature with iterative versions of neighborhood filters. However, for these 
applications, both images of the pair must be correctly and precisely registered. 
The iteration of the neighborhood filter was not originally considered by the 
pioneering works of Lee and Yaroslavsky. However, recent applications have shown 
its interest. The iteration of the filter as a local smoothing operator tends to piecewise 
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Fig. 4 Residual differences between original and filtered images in Fig. 3. For moderated values of 
h this residual contains details and texture but it doesn’t contain contrasted edges. These contrasted 
information is removed by the bilateral filter only for large values of h 


constant images by creating artificial discontinuities in regular zones. Barash et al. 
[6] showed that an iteration of the neighborhood filter was equivalent to a step of a 
certain numerical scheme of the classical Perona-Malik equation [57]. A complete 
proof of the equivalence between the neighborhood filter and the Perona-Malik 
equation was presented in [13] including a modification of the filter to avoid the 
creation of shocks inside regular parts of the image. Another theoretical explanation 
of the shock effect of the neighborhood filters can be found in Van de Weijer and van 
den Boomgaard [71] and Comaniciu [20]. Both papers show that the iteration of the 
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Fig. 5 Colorization experiment using the linear iteration of the neighborhood filter. Top left: input 
image with original luminance and initial data on the chromatic components. Bottom right: result 
image by applying the linear neighborhood scheme to the chromatic components using the initial 
chromatic data as boundary conditions. Top middle and right: initial data on the two chromatic 
components. Bottom middle and bottom right: final interpolated chromatic components 


neighborhood filter process makes points tend to the local modes of the histogram 
but in a different framework: the first for images and the second for any dimensional 
clouds of points. This discontinuity or shock creation in regular zones of the image 
is not desirable for filtering or denoising applications. However, it can be used for 
image or video editing as proposed by Winnemoller et al. [73] in order to simplify 
video content and achieve a cartoon look. 

Even if it may seem paradoxical, linear schemes have showed to be more 
useful than nonlinear ones for iterating the neighborhood filter; that is, the weight 
distribution for each pixel is computed once and is maintained during the whole 
iteration process. We will show in Sect. 4 that by computing the weights on an image 
and keeping them constant during the iteration process, a histogram concentration 
phenomenon makes the filter a powerful segmentation algorithm. The same iteration 
is useful to linearly diffuse or filter any initial data or seeds as proposed by Grady 
et al. [38] for medical image segmentation or [11] for colorization (see Fig.5 for 
an example). The main hypothesis for this seed diffusion algorithm is that pixels 
having a similar gray level value should be related and are likely to belong to the 
same object. Thus, pixels of different sites are related as in a graph with a weight 
depending on the gray level distance. The iteration of the neighborhood filter on the 
graph is equivalent to the solution of the heat equation on the graph by taking the 
graph Laplacian. Eigenvalues and eigenvectors of such a graph Laplacian can be 
computed allowing the design of Wiener and thresholding filters on the graph (see 
[69] and [59, 60] for more details). 

Both the neighborhood filter and the NL-means have been adapted and extended 
for other types of data and other image-processing tasks: for 3D data set points 
[17, 26, 35, 43, 79], and [42]; demosaicking, the operation which transforms the “R 


Local Smoothing Neighborhood Filters 1607 


or G or B” raw image in each camera into an “R and G and B” image [15, 51, 
63]; movie colorization, [34] and [49]; image inpainting by proposing a nonlocal 
image inpainting variational framework with a unified treatment of geometry and 
texture [2] (see also [74]); zooming by a fractal-like technique where examples are 
taken from the image itself at different scales [29]; movie flicker stabilization [24], 
compensating spurious oscillations in the colors of successive frames; and super- 
resolution, an image zooming method by which several frames from a video, or 
several low-resolution photographs, can be fused into a larger image [62]. The main 
point of this super-resolution technique is that it gives up an explicit estimate of 
the motion, allowing actually for a multiple motion, since a block can look like 
several other blocks in the same frame. The very same observation is made in [30] 
for devising a super-resolution algorithm and in [22, 33]. 


2 Denoising 
Analysis of Neighborhood Filter as a Denoising Algorithm 


In this section, we will further investigate the neighborhood filter behavior as a 
denoising algorithm. We will consider the simplest neighborhood filter version 
which averages spatially close pixels with an intensity difference lower than a 
certain threshold /. By classical probability theory, the average of N random and 
i.i.d values has a variance N times smaller than the variance of the original values. 
However, this theoretical reduction is not observed when applying neighborhood 
filters. 

In order to evaluate the noise reduction capability of the neighborhood filter, we 
apply it to a noise sample and evaluate the variance of the filtered sample. Let us 
suppose that we observe the realization of a white noise at a pixel x,n(x) = a. 
The nearby pixels with an intensity difference lower than h will be independent and 
identically distributed with the probability distribution function the restriction of the 
Gaussian to the interval (a — h,a + h). If the research zone is large enough, then 
the average value will tend to the expectation of such a variable. Thus, the increase 
of the research zone and therefore of the number of pixels being averaged does not 
increase the noise reduction capability of the filter. Such a noise reduction factor is 
computed in the next result. 


Theorem 1. Assume that the n(i) are i.i.d. with zero mean and variance o2. Then, 
the filtered noise by the neighborhood filter NF», satisfies the following: 


¢ The noise reduction depends only on the value of h, 


Var NFan(x) = f (5) a", 
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0.2 


1 2 3 4 5 


Fig. 6 Noise reduction function f(x) given by Theorem | 


where 


- 1 1 2xa 2 (a+x)* = 
fo = on | aay Ct Wee F da 


is a decreasing function with f (0) = 1 and lim, f(x) = 0. 
¢ The values NF,n(x) and NF,n(y) are uncorrelated for x # y. 


The function f(x) is plotted in Fig. 6. The noise reduction increases as the ratio 
h/o also does. We see that f(x) is near zero for values of x over 2.5 or 3, that 
is, values of h over 2.50 or 30, which justifies the values proposed in the original 
papers by Lee and Yaroslavsky. However, for a Gaussian variable, the probability of 
observing values at a distance of the average larger than 2.5 or 3 times the standard 
deviation is very small. Thus, by taking these values, we excessively increase the 
probability of mismatching pixels of different objects. Thus, close objects with an 
intensity contrast lower than 30 will not be correctly denoised. This explains the 
decreasing performance of the neighborhood filter as the noise standard deviation 
increases. 


The previous theorem also tells us that the denoised noise values are still 
uncorrelated once the filter has been applied. This is easily justified since we showed 
that as the size p of the neighborhood increases, the filtered value tends to the 
expectation of the Gauss distribution restricted to the interval (n(x) — h,n(x) + A). 
The filtered value is therefore a deterministic function of n(x) and h. Independent 
random variables are mapped by a deterministic function on independent variables. 

This property may seem anecdotic since noise is what we wish to get rid of. Now, 
it is impossible to totally remove noise. The question is how the remnants of noise 
look like. The transformation of a white noise into any correlated signal creates 
structure and artifacts. Only white noise is perceptually devoid of structure, as was 
pointed out by Attneave [3]. 
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The only difference between the neighborhood filter and the bilateral or SUSAN 
filter is the way the spatial component is treated. While for the classical neighbor- 
hood all pixels within a certain distance are treated equally, for the bilateral filter, 
pixels closer to the reference pixel are more important. Even if this can seem a 
slight difference, this is crucial from a qualitative point of view, that is, the creation 
of artifacts. 

It is easily shown that introducing the weighting function on the intensity 
difference instead of a non-weighted average does not modify the second property of 
Theorem |, and the denoised noise values are still uncorrelated if p is large enough. 
However, the introduction of the spatial kernel by the bilateral or SUSAN filter 
affects this property. Indeed, the introduction of a spatial decay of the weights makes 
denoised values at close positions to be correlated. 

There are two ways to show how denoising algorithms behave when they are 
applied to a noise sample. One of them is to find a mathematical proof that the pixels 
remain independent (or at least uncorrelated) and identically distributed random 
variables. The experimental device simply is to observe the effect of denoising 
on the simulated realization of a white noise. Figure 7 displays the filtered noises 
for the neighborhood filter, the bilateral filter, and other state-of-the-art denoising 
algorithms. 


Neighborhood Filter Extension: The NL-Means Algorithm 


Now in a very general sense inspired by the neighborhood filter, one can define as 
“neighborhood of a pixel x” any set of pixels y in the image such that a window 
around y looks like a window around x. All pixels in that neighborhood can be used 
for predicting the value at x, as was shown in [23,31] for texture synthesis and in [21, 
80] for inpainting purposes. The fact that such a self-similarity exists is a regularity 
assumption, actually more general and more accurate than all regularity assumptions 
we consider when dealing with local smoothing filters, and it also generalizes a 
periodicity assumption of the image. 

Let v be the noisy image observation defined on a bounded domain 2 C R?, and 
let x € §2. The NL-means algorithm estimates the value of x as an average of the 
values of all the pixels whose Gaussian neighborhood looks like the neighborhood 
of x: 


1 (Gax vo+)—vy +?) 0) 
NLON®) = a5 ie e P v(y) dy, 3) 


where G, is a Gaussian kernel with standard deviation a, hf acts as a filtering 
(Gax\va+)J—ve+.)|) 0) 

parameter, and C(x) = / 2e dz is the normalizing factor. We 

recall that 


(Gee ver + )- v0 + IP) O =f Game +) - vO + OP ar, 
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Fig. 7 The noise to noise criterion. From left to right and from top to bottom: original noise image 
of standard deviation 20, Gaussian convolution, anisotropic filtering, total variation, TIHWT, DCT 
empirical Wiener filter, neighborhood filter, bilateral filter, and the NL-means. Parameters have 
been fixed for each method so that the noise standard deviation is reduced by a factor 4. The filtered 
noise by the Gaussian filter and the total variation minimization are quite similar, even if the first 
one is totally blurred and the second one has created many high frequency details. The filtered 
noise by the hard wavelet thresholding looks like a constant image with superposed wavelets. The 
filtered noise by the neighborhood filter and the NL-means algorithm looks like a white noise. This 
is not the case for the bilateral filter, where low frequencies of noise are enhanced because of the 
spatial decay 


We will see that the use of an entire window around the compared points makes this 
comparison more robust to noise. For the moment, we will compare the weighting 
distributions of both filters. Figure8 illustrates how the NL-means algorithm 
chooses in each case a weight configuration adapted to the local geometry of the 
image. Then, the NL-means algorithm seems to provide a feasible and rational 
method to automatically take the best of all classical denoising algorithms, reducing 
for every possible geometric configuration the mismatched averaged pixels. It 
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dead 

a 
; Ree 
Fig. 8 Weight distribution of NL-means, the bilateral filter, and the anisotropic filter used to 
estimate the central pixel in four detail images. On the two right-hand-side images of each triplet, 
we display the weight distribution used to estimate the central pixel of the left image by the 
neighborhood and the NL-means algorithm. (a) In straight edges, the weights are distributed in the 
direction of the level line (as the mean curvature motion). (b) On curved edges, the weights favor 
pixels belonging to the same contour or level line, which is a strong improvement with respect 
to the mean curvature motion. In the cases of (c) and (d), the weights are distributed across the 


more similar configurations, even though they are far away from the observed pixel. This shows a 
behavior similar to a nonlocal neighborhood filter or to an ideal Wiener filter 


preserves flat zones as the Gaussian convolution and straight edges as the anisotropic 
filtering while still restoring corners or curved edges and texture. 

Due to the nature of the algorithm, one of the most favorable cases is the textural 
case. Texture images have a large redundancy. For each pixel, many similar samples 
can be found in the image with a very similar configuration, leading to a noise 
reduction and a preservation of the original image. In Fig. 9, one can see an example 
with a Brodatz texture. The Fourier transform of the noisy and restored images 
shows the ability of the algorithm to preserve the main features even in the case of 
high frequencies. 

The NL-means seems to naturally extend the Gaussian, anisotropic, and neigh- 
borhood filtering. But it is not easily related to other state-of-the-art denoising 
methods as the total variation minimization [64], the wavelet thresholding [19, 27], 
or the local DCT empirical Wiener filters [76]. For this reason, we compare 
these methods visually in artificial denoising experiences (see [12] for a more 
comprehensive comparison). 

Figure 10 illustrates the fact that a nonlocal algorithm is needed for the 
correct reconstruction of periodic images. Local smoothing filters and Wiener and 
thresholding methods are not able to reconstruct the wall pattern. Only NL-means 
and the global Fourier-Wiener filter reconstruct the original texture. The Fourier- 
Wiener filter is based on a global Fourier transform, which is able to capture the 
periodic structure of the image in a few coefficients. But this only is an ideal 
filter: the Fourier transform of the original image is being used. Figure 8d shows 
how NL-means chooses the correct weight configuration and explains the correct 
reconstruction of the wall pattern. 

The NL-means algorithm is not only able to restore periodic or texture Images; 
natural images also have enough redundancy to be restored. For example, in a 
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Fig. 9 NL-means denoising experiment with a Brodatz texture image. Left: noisy image with 
standard deviation 30. Right: NL-means restored image. The Fourier transforms of the noisy and 
restored images show how main features are preserved even at high frequencies 
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Fig. 10 Denoising experience on a periodic image. From left to right and from top to bottom: 
noisy image (standard deviation 35), Gauss filtering, total variation, neighborhood filter, Wiener 
filter (ideal filter), TIHWT (translation invariant hard thresholding), DCT empirical Wiener 
filtering, and NL-means 
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Fig. 11 Denoising experience on a natural image. From left to right and from top to bottom: 
noisy image (standard deviation 35), total variation, neighborhood filter, translation invariant hard 
thresholding (TIHWT), empirical Wiener, and NL-means 


flat zone, one can find many pixels lying in the same region and with similar 
configurations. In a straight or curved edge, a complete line of pixels with a similar 
configuration is found. In addition, the redundancy of natural images allows us to 
find many similar configurations in faraway pixels. 

Figure 11 shows that wavelet and DCT thresholding are well adapted to the 
recovery of oscillatory patterns. Although some artifacts are noticeable in both 
solutions, the stripes are well reconstructed. The DCT transform seems to be more 
adapted to this type of texture, and stripes are a little better reconstructed. For a 
much more detailed comparison between sliding window transform domain filtering 
methods and wavelet threshold methods, we refer the reader to [77]. NL-means also 
performs well on this type of texture, due to its high degree of redundancy. 

The above description of movie denoising algorithms and its juxtaposition to 
the NL-means principle shows how the main problem, motion estimation, can be 
circumvented. In denoising, the more samples we have the happier we are. The 
aperture problem is just a name for the fact that there are many blocks in the next 
frame similar to a given one in the current frame. Thus, singling out one of them in 
the next frame to perform the motion compensation is an unnecessary and probably 
harmful step. A much simpler strategy that takes advantage of the aperture problem 
is to denoise a movie pixel by involving indiscriminately spatial and temporal 
similarities (see [14] for more details on this discussion). The algorithm favors 
pixels with a similar local configuration, as the similar configurations move, so do 
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the weights. Thus, the algorithm is able to follow the similar configurations when 
they move without any explicit motion computation (see Fig. 12). 


Extension to Movies 


Averaging filters are easily extended to the denoising of image sequences and video. 
The denoising algorithms involve indiscriminately pixels not belonging only to the 
same frame but also the previous and posterior ones. 

In many cases, this straightforward extension cannot correctly deal with moving 
objects. For that reason, state-of-the-art movie filters are motion compensated 
(see [10] for a comprehensive review). The underlying idea is the existence of 
a “ground true” physical motion, which motion estimation algorithms should be 
able to estimate. Legitimate information should exist only along these physical 
trajectories. The motion compensated filters estimate explicitly the motion of the 
sequence by a motion estimation algorithm. The motion compensated movie yields 
a new stationary data on which an averaging filter can be applied. The motion 
compensation movie yields a new stationary data on which an averaging filter can 
be applied. The motion compensation neighborhood filter was proposed by Ozkan 
et al. [55]. We illustrate in Fig. 13 the improvement obtained with the proposed 
compensation. 

One of the major difficulties in motion estimation is the ambiguity of trajectories, 
the so-called aperture problem. This problem is illustrated in Fig. 14. At most pixels, 
there are several options for the displacement vector. All of these options have a 
similar gray level value and a similar block around them. Now, motion estimators 
have to select one by some additional criterion. 


3 Asymptotic 
PDE Models and Local Smoothing Filters 


According to Shannon’s theory, a signal can be correctly represented by a discrete 
set of values, the “samples,” only if it has been previously smoothed. Let us start 
with uo the physical image, a real function defined on a bounded domain 2 C 
IR*. Then a blur optical kernel k is applied, i.e., uo is convolved with k to obtain 
an observable signal k*uo. Gabor remarked in 1960 that the difference between 
the original and the blurred images is roughly proportional to its Laplacian, Au = 
Uxx + Uyy. In order to formalize this remark, we have to notice that k is spatially 
concentrated and that we may introduce a scale parameter for k, namely, k;,(x) = 


hk (atx), If, for instance, u is C* and bounded and if k is a radial function in 
the Schwartz class, then 
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Fig. 12 Weight distribution of NL-means applied to a movie. In (a), (b), and (c), the first row 
shows a five frames image sequence. In the second row, the weight distribution used to estimate 
the central pixel (in white) of the middle frame is shown. The weights are equally distributed 
over the successive frames, including the current one. They actually involve all the candidates for 
the motion estimation instead of picking just one per frame. The aperture problem can be taken 


advantage of for a better denoising performance by involving more pixels in the average Fig. 
displays the application of the denoising methods to a white noise. We display the filtered noise 
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Fig. 13 Comparison of static filters, motion compensated filters, and NL-means applied to an 
image sequence. Top: three frames of the sequence are displayed. Middle and left to right: 
neighborhood filter, motion compensated neighborhood filter, and the NL-means. (AWA). Bottom: 
the noise removed by each method (difference between the noisy and filtered frame). Motion 
compensation improves the static algorithms by better preserving the details and creating less blur. 
We can read the titles of the books in the noise removed by AWA. Therefore, that much information 
has been removed from the original. Finally, the NL-means algorithm (bottom row) has almost no 
noticeable structure in its removed noise. As a consequence, the filtered sequence has kept more 
details and is less blurred 


UoxKn (x) — uo(X) 
h 


=> cAuo(x). 


Hence, when h gets smaller, the blur process looks more and more like the heat 
equation 


u; = cAu, u(0) — uo. 
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Fig. 14 Aperture problem and the ambiguity of trajectories are the most difficult problems in 
motion estimation: There can be many good matches. The motion estimation algorithms must pick 
one 


Thus, Gabor established a first relationship between local smoothing operators and 
PDEs. The classical choice for k is the Gaussian kernel. 

Remarking that the optical blur is equivalent to one step of the heat equation, 
Gabor deduced that we can, to some extent, deblur an image by reversing the time 
in the heat equation, uw, = —Au. Numerically, this amounts to subtracting the filtered 
version from the original image: 


u— Gyxu = —h? Au + o(h’). 


This leads to considering the reverse heat equation as an image restoration, ill-posed 
though it is. The time-reversed heat equation was stabilized in the Osher-Rudin 
shock filter [54] who proposed 


u, = —sign(L(u)) | Du, 


where the propagation term |Du| is tuned by the sign of an edge detector A(u). The 
function A(u) changes sign across the edges where the sharpening effect therefore 
occurs. In practice, A(u) = Au and the equation is related to a reverse heat equation. 

The early Perona-Malik “anisotropic diffusion” [57] is directly inspired from the 
Gabor remark. It reads 


u, = div (s ({Du/?) Du), 


where g : [0, +00) — [0, +00) is a smooth decreasing function satisfying g(0) = 
1, lim;++400 g(s) = 0. This model is actually related to the preceding ones. Let us 
consider the second derivatives of u in the directions of Du and Dut: 


Du D Dut Dut 
Unyn = Du aceasta , Ugg = Du : Rss . 
|Du| |Du| |Du| |Du| 
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Then, Eq. (5) can be rewritten as 
uy = 8 (|Du\*) Use th (|Du)?) Unn 


where h(s) = g(s) + 2sg’(s). Perona and Malik proposed the function g(s) = 
Tak: In this case, the coefficient of the first term is always positive and this term 
therefore appears as a one-dimensional diffusion term in the orthogonal direction 
to the gradient. The sign of the second coefficient, however, depends on the value 
of the gradient. When |Du|* < k, this second term appears as a one-dimensional 
diffusion in the gradient direction. It leads to a reverse heat equation term when 
|Du|* > k. 

The Perona-Malik model has got many variants and extensions. Tannenbaum 
and Zucker [45] proposed, endowed in a more general shape analysis framework, 
the simplest equation of the list: 


; Du 
uy = | Du| div ]Dul = Ug. 


This equation had been proposed some time before in another context by Sethian 
[66] as a tool for front propagation algorithms. This equation is a “pure” diffusion 
in the direction orthogonal to the gradient and is equivalent to the anisotropic filter 
AF [40]: 


AFpu(x) = f Gateute + 18)ar, 


where € = Du(x)+/|Du(x)| and G;,(t) denotes the one-dimensional Gauss function 
with variance h?. 

This diffusion is also related to two models proposed in image restoration. The 
Rudin-Osher-Fatemi [64] total variation model leads to the minimization of the total 
variation of the image TV(u) = f |Du|, subject to some constraints. The steepest 
descent of this energy reads, at least formally, 


i ee (4) 
ot | Du 


which is related to the mean curvature motion and to the Perona-Malik equation 


i 
|Du| 


again (4). An existence and uniqueness theory is available for this equation [1]. 


when g (\Dul*) = . This particular case, which is not considered in [57], yields 
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Asymptotic Behavior of Neighborhood Filters (Dimension 1) 


Let u denote a one-dimensional signal defined on an interval J C R and consider 
the neighborhood filter 


|u(y)—u(x )/? 


x+ 
NF) pu(x) = as aa je Pd 
h,p C(x) y ys 
x—=p 


ee = [woy—wool? 
where C(x) = f, ang: 

The following theorem describes the asymptotical behavior of the neighborhood 
filter in 1D. The proof of this theorem and next ones in this section can be found in 
[13]. 


Theorem 2. Suppose u € C?(I), and let p,h,a > 0 such that p,h > 0 a 
h = 0(p*). Consider the continuous function g(t) = a fort #0, g(0) = 
where E(t) = 2 é —" ds. Let f be the continuous function 


70 = 89 + H- 5, £0 = 


Then, for x € R, 


1. Ife <1, NFppu(x) — u(x) ~ “2p? 
2. Ife = 1, NFypu(x) — u(x) =~ f (F |w/(x)|) u(x) p?. 
3. If 1 <a < 3,NFrpu(x) — u(x) ~ g (p!® [u/(x)|) u(x) 6°. 


According to Theorem 2, the neighborhood filter makes the signal evolve propor- 
tionally to its second derivative. The equation u, = cu’ acts as a smoothing or 
enhancing model depending on the sign of c. Following the previous theorem, we 
can distinguish three cases depending on the values of h and p. First, if is much 
larger than p, the second derivative is weighted by a positive constant and the signal 
is therefore filtered by a heat equation. Second, if h and p have the same order, the 
sign and magnitude of the weight is given by f ( P |u'(x) |). As the function f takes 
positive and negative values (see Fig. 15), the filter behaves as a filtering/enhancing 
algorithm depending on the magnitude of |u'(x)|. If B denotes the zero of f, then 
a filtering model is applied wherever |u'| < Bt and an enhancing model wherever 


|u'| > B 2 The intensity of the enhancement tends to zero when the derivative tends 
to infinity. Thus, points x where |u’(x)| is large are not altered. The transition of the 
filtering to the enhancement model creates a singularity in the filtered signal. In the 
last case, p is much larger than / and the sign and magnitude of the weight is given 
by g (¢ |u (x)|). Function g is positive and decreases to zero. If the derivative of u 
is bounded, then ? |u'(x)| tends to infinity and the intensity of the filtering to zero. 
In this case, the signal is hardly modified. 
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Fig. 15 One dimensional neighborhood filter experiment. The neighborhood filter is iterated until 
the steady state is attained for different values of the ratio p/h. Top: Original sine signal. Middle 
left: filtered signal with p/h = 10~°. Middle right: filtered signal with p/h = 108. Bottom 
left: filtered signal with p/h = 2. Bottom right: filtered signal with p/h = 5. The examples 
corroborate the results of Theorem 2. If p/h tends to zero the algorithm behaves like a heat 
equation and the filtered signal tends to a constant. If, instead, p/h tends to infinity the signal 
is hardly modified. If p and h have the same order, the algorithm presents a filtering/enhancing 
dynamic. Singularities are created due to the transition of smoothing to enhancement. The number 
of enhanced regions strongly depends upon the ratio . as illustrated in the bottom figures 


In summary, a neighborhood filter in dimension I shows interesting behavior 
only if p and h have the same order of magnitude, in which case the neighborhood 
filter behaves like a Perona-Malik equation. It enhances edges with a gradient above 
a certain threshold and smoothes the rest. 

Figure 16 illustrates the behavior of the one-dimensional neighborhood filter. The 
algorithm is iterated until the steady state is attained on a sine signal for different 
values of the ratio p/h. The results of the experiment corroborate the asymptotical 
expansion of Theorem 2. In the first experiment, p/h = 10~* and the neighborhood 
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Fig. 16 One-dimensional neighborhood filter experiment. The neighborhood filter is iterated until 
the steady state is attained for different values of the ratio p/h. Top: original sine signal. Middle 
left: filtered signal with p/h = 10~®. Middle right: filtered signal with p/h = 108. Bottom left: 
filtered signal with p/h = 2. Bottom right: filtered signal with p/h = 5. The examples corroborate 
the results of Theorem 2. If o/h tends to zero, the algorithm behaves like a heat equation and the 
filtered signal tends to a constant. If, instead, p/h tends to infinity, the signal is hardly modified. 
If p and h have the same order, the algorithm presents a filtering/enhancing dynamic. Singularities 
are created due to the transition of smoothing to enhancement. The number of enhanced regions 
strongly depends upon the ratio ie as illustrated in the bottom figures 


filter is equivalent to a heat equation. The filtered signal tends to a constant. In the 
second experiment, p/h = 108 and the value g (5 |u! |) is nearly zero. As predicted 
by the theorem, the filtered signal is nearly identical to the original one. The last 
two experiments illustrate the filtering/enhancing behavior of the algorithm when h 
and p have similar values. As predicted, an enhancing model is applied where the 
derivative is large. Many singularities are being created because of the transition 
of the filtering to the enhancing model. Unfortunately, the number of singularities 
and their position depend upon the value of p/h. This behavior is explained by 
Theorem 2(2). Figure 22 illustrates the same effect in the 2D case. 

The filtering/enhancing character of the neighborhood filter is very different 
from a pure enhancing algorithm like the Osher-Rudin shock filter. Figures 17 
and 18 illustrate these differences. In Fig. 17, the minimum and the maximum of 
the signal have been preserved by the shock filter, while these two values have been 
significantly reduced by the neighborhood filter. This filtering/enhancing effect is 
optimal when the signal is noisy. Figure 18 shows how the shock filter creates 
artificial steps due to the fluctuations of noise, while the neighborhood filter reduces 
the noise avoiding any spurious shock. Parameter / has been chosen larger than 
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Fig. 17 Comparison between the neighborhood filter and the shock filter. Top: original signal. 
Bottom left: application of the neighborhood filter. Bottom right: application of the shock filter. The 
minimum and the maximum of the signal have been preserved by the shock filter and reduced by 
the neighborhood filter. This fact illustrates the filtering/enhancing character of the neighborhood 
filter compared with a pure enhancing filter 
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Fig. 18 Comparison between the neighborhood filter and the shock filter. Top: original signal. 
Bottom left: application of the neighborhood filter. Bottom right: application of the shock filter. 
The shock filter is sensitive to noise and creates spurious steps. The filtering/enhancing character 
of the neighborhood filter avoids this effect 


the amplitude of noise in order to remove it. Choosing an intermediate value of h, 
artificial steps could also be generated on points where the noise amplitude is above 
this parameter value. 
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The Two-Dimensional Case 
The following theorem extends the previous result to the two-dimensional case. 


Theorem 3. Suppose u € C*(Q), and let p,h,a > O such that p,h — 0 and 


2 
1 te! 
3 3E(t)’ 


fort £0,g(0) = i, where E(t) = 21, eds. Let f be the continuous function 
defined by 


h = O(p%). Let us consider the continuous function g defined by g(t) = 


oe ae 3g(t) 1 ae 
Then, for x € 92, 
1. Ifa <1, 
Au(x 
NF), pu(x) — u(x) ~ ae) 
2. Ifa = 1, 


NF}, pit(x2) — u(x) © [ (fC |Du(x)|) uee(x) + f (F |Du(x)1) tin 2] 0. 
SHiew 22 


NF) pu(x) — u(x) & & (eo! |Du(x)|) [wee(x) + 3uyn(x)] p- 
where € = Du(x)+/|Du(x)| and n = Du(x)/|Du(x)|. 


According to Theorem 3, the two-dimensional neighborhood filter acts as an 
evolution PDE with two terms. The first term is proportional to the second derivative 
of u in the direction € = Du(x)+/|Du(x)|, which is tangent to the level line 
passing through x. The second term is proportional to the second derivative of u 
in the direction 7 = Du(x)/|Du(x)|, which is orthogonal to the level line passing 
through x. Like in the one-dimensional case, the evolution equations u, = c,ug¢ and 
Uy; = C2Uy, act as filtering or enhancing models depending on the signs of c, and c3. 
Following the previous theorem, we can distinguish three cases, depending on the 
values of h and p. 

First, if h is much larger than p, both second derivatives are weighted by the 
same positive constant. Thus, the sum of both terms is equivalent to the Laplacian 
of u, Au, and we get back to Gaussian filtering. 

Second, if 4 and p have the same order of magnitude, the neighborhood filter 
behaves as a filtering/enhancing algorithm. The coefficient of the diffusion in the 
tangential direction, ugg, is given by g (¢ |Dul). The function g is positive and 
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decreasing. Thus, there is always diffusion in that direction. The weight of the 
normal diffusion, uy,, is given by f ( p |Dul). As the function f takes positive and 
negative values (see Fig. 15), the filter behaves as a filtering/enhancing algorithm 
in the normal direction and depending on |Du|. If B denotes the zero of f, then a 
filtering model is applied wherever | Du| < B A and an enhancing strategy wherever 


|Du| > B“. The intensity of the filtering in the tangent diffusion and the enhancing 
in the normal diffusion tend to zero when the gradient tends to infinity. Thus, points 
with a very large gradient are not altered. 

Finally, if o is much larger than h, the value P tends to infinity and then the 
filtering magnitude g (2 | Dul) tends to zero. Thus, the original image is hardly 
altered. Let us mention that similar calculations were performed in a particular case 
for the neighborhood median filter by Masnou [52]. 

We observe that when p and / have the same order, the neighborhood filter 
asymptotically behaves like a Perona-Malik model. Let us be more specific about 


1 
this comparison. Taking g(s) = g (s?) in the Perona-Malik Eq. (6), we obtain 


uy = &(|Dul) uge +h (|Dul) yy, (5) 


where h(s) = £(s) + sg'(s). Thus, the Perona-Malik model and the neighborhood 
filter can be decomposed in the same_way and with exactly the same weight in the 
tangent direction. Then the function h has the same behavior as f (Theorem 3), as 
can be observed in Fig. 19. Thus, in this case, a neighborhood filter has the same 
qualitative behavior as a Perona-Malik model, even if we cannot rewrite it exactly 
as such. 


Fig. 19 Weight comparison of the neighborhood filter and the Perona-Malik equation. Magnitude 
of the tangent diffusion (continuous line, identical for both models) and normal diffusion (dashed 
line- -) of Theorem 3. Magnitude of the tangent diffusion (continuous line) and normal diffusion 
(dashed line- - -) of the Perona-Malik model (5). Both models show nearly the same behavior 
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Figure 22 displays a comparison of the neighborhood filter and the Perona-Malik 
model. We display a natural image and the filtered images by both models. These 
solutions have a similar visual quality and tend to display flat zones and artificial 
contours inside the smooth regions. Figure 23 corroborates this visual impression. 
We display the level lines of both filtered solutions. As expected from the above 
consistency theorems, for both models the level lines of the original image tend to 
concentrate, thus creating large flat zones separated by edges. The solutions are very 
close, up to the obvious very different implementations. The neighborhood filter is 
implemented exactly as in its definition and the Perona-Malik model by the explicit 
difference scheme proposed in the original paper. 


A Regression Correction of the Neighborhood Filter 


In the previous sections, we have shown the enhancing character of the neigh- 
borhood filter. We have seen that the neighborhood filter, like the Perona-Malik 
model, can create large flat zones and spurious contours inside smooth regions. 
This effect depends upon a gradient threshold which is hard to fix in such a way 
as to always separate the visually smooth regions from edge regions. In order to 
avoid this undesirable effect, let us analyze in more detail what happens with the 
neighborhood filter in the one-dimensional case. 

Figure 20 shows a simple illustration of this effect. For each x in the convex part 
of the signal, the filtered value is the average of the points y such that u(x) —h < 
u(y) < u(x) + A for a certain threshold h. As it is illustrated in the figure, the 
number of points satisfying u(x) —h < u(y) < u(x) is larger than the number 
of points satisfying u(x) < u(y) < u(x) + h. Thus, the average value YNF(x) 
is smaller than u(x), enhancing this part of the signal. A similar argument can be 
applied in the concave parts of the signal, dealing with the same enhancing effect. 
Therefore, shocks will be created inside smooth zones where concave and convex 
parts meet. Figure 20 also shows how the mean is not a good estimate of u(x) 
in this case. In the same figure, we display the regression line approximating u 


Fig. 20 Illustration of the 
shock effect of the YNF on 
the convex of a signal. The 
number of points y satisfying 
u(x) —h < u(y) < u(x) is 
larger than the number 
satisfying 

u(x) < u(y) < u(x) +h. 
Thus, the average value 
YNF(x) is smaller than u(x), 
enhancing that part of the 
signal. The regression line of 
u inside (x_, x) better 
approximates the signal at x 
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inside (u~'(u(x) — h),u~!(u(x) + h)). We see how the value of the regression 
line at x better approximates the signal. In the sequel, we propose to correct the 
neighborhood filter with this better estimate. 

In the general case, this linear regression strategy amounts to finding for every 
point x the plane locally approximating u in the following sense: 


_ |u| 


min f w(x, y)(u(y) —aiy1—ao)dy,  w(x,y) =e P 
Bo(x) 


d0,a] 


and then replacing u(x) by the filtered value a,x; + do. The weights used to define 
the minimization problem are the same as the ones used by the neighborhood 
filter. Thus, the points with a gray level value close to u(x) will have a larger 
influence in the minimization process than those with a further gray level value. 
We denote the above linear regression correction by LNF,,,. Taking a} = 0 and 
then approximating uw by a constant function, the minimization (5) goes back to the 
neighborhood filter. 

This minimization was originally proposed by Cleveland [18] with a weight 
family not depending on the function u but only on the spatial distance of x and 
y. A similar scheme incorporating u in the weight computation has been statistically 
studied in [61]. The authors propose an iterative procedure that describes for 
every point the largest possible neighborhood in which the initial data can be well 
approximated by a parametric function. 

Another similar strategy is the interpolation by ENO schemes [41]. The goal of 
ENO interpolation is to obtain a better adapted prediction near the singularities of 
the data. For each point it selects different stencils of fixed size M and for each 
stencil reconstructs the associated interpolation polynomial of degree M. Then the 
least oscillatory polynomial is selected by some prescribed numerical criterion. The 
selected stencils tend to escape from large gradients and discontinuities. 

The regression strategy also tends to select the right points in order to approxi- 
mate the function. Instead of choosing a certain interval, all the points are used in 
the polynomial reconstruction, but weighted by the gray level differences. 

As in the previous sections, let us analyze the asymptotic behavior of the linear 
regression correction. We compute the asymptotic expansion of the filter when 0 < 
a < 1. We showed that when a > 1, the signal is hardly modified. 

For the sake of completeness, we first compute the asymptotic expansion in the 
one-dimensional case. 


Theorem 4. Suppose u € C?(1), and let p,h,a > 0 such that p,h — 0 and 
h = O(p*). Let f be the continuous function defined as f (0) = z, 


s 1 Ite * 
fo ge(1- 3). 


fort #0, where E(t) = 2 fy e~* ds. Then, for x € R, 
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Fig. 21 Weighting functions of Theorems 4 and 5. Left: function f of Theorem 4. Right: constant 
function 1/6 (continuous line) and function f (dashed line) of Theorem 5 


1. If <1, LNFy, pu(x) — u(x) = Ep. 
2. Ife = 1,NFhpu(x) — u(x) = f (2 [u! (x)|) w(x) p?. 


Theorem 4 shows that the LNFy,, filter lets the signal evolve proportionally to its 
second derivative, as the neighborhood filter does. When / is larger than p, the 
filter is equivalent to the original neighborhood filter and the signal is filtered by a 
heat equation. When p and / have the same order, the sign and magnitude of the 
filtering process is given by 7 (2 |u' (x) l) (see Fig. 21). This function is positive and 
quickly decreases to zero. Thus, the signal is filtered by a heat equation of decreasing 
magnitude and is not altered wherever the derivative is very large. 
The same asymptotic expansion can be computed in the two-dimensional case. 


Theorem 5. Suppose u € C?(Q), and let p,h,a > 0 such that p,h — 0 and 
h = O(p*). Let f be the continuous function defined as f (0) = i, 


fort #0, where E(t) = oi. e~* ds. Then, forx € Q, 


1. Ifa <1, 


Au(x) 5 


LNF)p p(x) — u(x) & 6 


2. a =1, 


; 1 
LNF), pix) — u(x) ~ Ei (F |Du(x)|) Un (X)(X) + zee) ro 
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According to the previous theorem, the filter can be written as the sum of two 
diffusion terms in the direction of & and 7. When h is much larger than p, the 
linear regression correction is equivalent to the heat equation like the original 
neighborhood filter. When p and h have the same order, the behavior of the 
linear regression algorithm is very different from the original neighborhood filter. 
The function weighting the tangent diffusion is a positive constant. The function 
weighting the normal diffusion is positive and decreasing (see Fig. 21), and therefore 
there is no enhancing effect. The algorithm combines the tangent and normal 
diffusion wherever the gradient is small. Wherever the gradient is larger, the normal 
diffusion is canceled and the image is filtered only in its tangent direction. This 
subjacent PDE was already proposed as a diffusion equation in [4]. This diffusion 
makes the level lines evolve proportionally to their curvature. In the Perona-Malik 
model, the diffusion is stopped near the edges. In this case, the edges are filtered by 
a mean curvature motion. 

It may be asked whether the modified neighborhood filter still preserves signal 
discontinuities. The answer is yes. It is easily checked that for small enough h, all 
piecewise affine functions with smooth jump curves are steady. Thus, the behavior 
is the same as for the classical neighborhood filter. Our asymptotic analysis is of 
course not valid for such functions, but only for smooth functions. 

As a numerical scheme, the linear regression neighborhood filter allows the 
implementation of a mean curvature motion without the computation of gradients 
and orientations. When the gradient is small, the linear regression filter naturally 
behaves like the heat equation. This effect is introduced on typical schemes 
implementing the mean curvature motion. In flat zones, the gradient is not well 
defined and some kind of isotropic diffusion must be applied. Therefore, the linear 
regression neighborhood filter naturally extends the mean curvature motion and 
yields a stable numerical scheme for its computation, independent of gradient 
orientations. 

Figure 22 displays an experiment comparing the LNF;,,, with the neighborhood 
filter and the Perona-Malik equation. The linear correction does not create any 
contour or flat zone inside the smooth regions. Figure 23 displays the level lines 
of the previous experiment. The level lines of the LNF;,, are filtered by a mean 
curvature motion, and they do not get grouped creating flat zones. The same effect 
is illustrated in Fig. 24. 


The Vector-Valued Case 


Let u be a vector-valued function defined on a bounded domain 2 C R?2,u: 2 —> 
IR". The vector neighborhood filter can be written as 


_ le) =n09) |)? 
12 


u(y)e dy, 


1 
Argh C(x) JB, (x) 
p x 
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Fig. 22 Comparison experiment. Top left: original image. Top right: Perona-Malik filtered image. 
Bottom left: filtered image by the neighborhood filter. Bottom right: filtered image by the linear 
regression neighborhood filter. The neighborhood filter experiments are performed by iterating the 
discrete version of definitions (1) and (5). Both the neighborhood filter and its linear regression 
correction have been applied with the same value of and p. The displayed images have been 
attained within the same number of iterations. The Perona-Malik equation is implemented by 
the explicit difference scheme proposed in the original paper. The Perona-Malik model and the 
neighborhood filter create artificial contours and flat zones. This effect is almost completely 
avoided by the linear regression neighborhood filter 


where ||u(y) — u(x)||? is now the Euclidean vector norm and each component func- 
tion u; is filtered with the same weight distribution. The linear regression correction 
is defined as in the scalar case, and each component is locally approximated by a 
plane with the same weight distribution. 

In order to compute the asymptotic expansion of the linear regression filter, we 
must fix a coordinate system for R?. In the scalar case, we used the reference system 
given by the gradient of the image at x and its orthogonal direction. In addition, this 
reference allows us to relate the obtained diffusion to the evolution of the level lines 
of the image and the mean curvature motion. Now, we cannot use the same reference 
and we need to define a new one. By analogy with the scalar case, we choose the 
directions of minimum and maximum variation of the vector function. 


Definition 1. We define the normal direction 7 and the tangent direction & as the 
vectors that respectively maximize and minimize the following variation: 
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Fig. 23 Level lines of the images in Fig. 22. By the Perona-Malik filter and the neighborhood 
filter, the level lines tend to group, creating flat zones. The regression correction filters the level 
lines by a curvature motion without creating any flat zone 


nN 
4 


Ou; 2 
Me | 


under the constraint ||v|| = 1. 
It is easily seen that this constrained optimization leads to the computation of the 
eigenvectors of the matrix 


Ena) 
ox ax? dy 
A = ’ 
au au\ |i du 
ax’ dy oy 
where 2 = ( % dun) and 2 = ( a4 dun). The two positive eigenvalues 
ax exer? ay dy dy? *? oy f° 


of A, A+ and A_, are the maximum and the minimum of the vector norm associated 
to A and the maximum and the minimum variations, as defined in Definition |. The 
corresponding eigenvectors are orthogonal leading to the above-defined normal and 
tangent directions. This orthonormal system was first proposed for vector-valued 
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Fig. 24 Comparison of the neighborhood filter and the linear regression correction. Top left: 
original image. Top middle: filtered image by the neighborhood filter. Top right: filtered image 
by the regression neighborhood filter. Bottom: level lines of a part of the images on the above line. 
Both neighborhood filters have been performed with the same filtering parameters and the same 
number of iterations. The linear regression neighborhood algorithm has filtered the image while 
preserving the main boundaries as the original neighborhood filter. No enhancing has been applied 
by the linear correction avoiding the shock effect. The level lines of the neighborhood filter tend to 
group and create large flat zones. In addition, these level lines oscillate, while those of the linear 
regression algorithm have been correctly filtered 


image analysis in [25]. Many PDE equations have been proposed for color image 
filtering using this system. We note the coherence-enhancing diffusion [72], the 
Beltrami flow [46], and an extension of the mean curvature motion [65]. 


Theorem 6. Suppose u € C?(2,R"), and let p,h,a > 0 such that p,h — 0 and 
h = O(p°). Let f be the continuous function defined as f (0) = i, 


fort 4 0, where E(t) = ai. eds. Then, for x € Q, 


1. Ifa <1, 


A 
u(x) 0. 


LNF)p p(x) — u(x) & 6 
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2. Ifa = 1, 

ou 
aw 


~(p || du 
F(Glae 


) Du(é.€)(x) 


) D*u(n, | 


where Au(x) = (Au;(x))\<j<n and Duly, v)(x) = Du; (v, v)(X)1<i<n for v € 
{n, &}. 


LNF;), u(x) — u(x) ~ f ( 


ae) 


Interpretation 

When / is much larger than p, the linear regression neighborhood filter is equivalent 
to the heat equation applied independently to each component. When / and p have 
the same order, the subjacent PDE acts as an evolution equation with two terms. 
The first term is proportional to the second derivative of u in the tangent direction 
&. The second term is proportional to the second derivative of u in the normal 
direction 7. The magnitude of each diffusion term depends on the variation in the 


respective direction, A_ = ECOI| and Ay = FECO| |. The weighting function 


f is positive and decreases to zero (see Fig. 21). We can distinguish the following 
cases depending on the values of A+ and A_. 


¢ IfAyz x A_ ~& O, then there are very few variations of the vector image u 
around x. In this case, the linear regression neighborhood filter behaves like a 
heat equation with maximum diffusion coefficient ry (0). 

¢ IfA4 >> A_, then there are strong variations of u around x and the point may be 
located on an edge. In this case, the magnitude Fa (PAs) tends to zero and there 
is no diffusion in the direction of maximal variation. If A_ >> 0, then x may be 
placed on an edge with different orientations depending on each component and 
the magnitude of the filtering in both directions tends to zero, so that the image 
is hardly altered. If A_ ~ 0, then the edges have similar orientations in all the 
components and the image is filtered by a directional Laplacian in the direction 
of minimal variation. 

¢ IfA, ~ A_ > O, then we may be located on a saddle point, and in this case the 
image is hardly modified. When dealing with multivalued images, one can think 
of the complementarity of the different channels leading to the perception of a 
corner. 


In the scalar case, the theorem gives back the result studied in the previous sections. 
The normal and tangent directions are, respectively, the gradient direction and the 
oe line direction. In this case, ae (X) = 0 and an *) = |Du(x)|, and we get 
ack to 


LNF, p(s) — us) = | ED? até, E900) + F (P [Ducs]) Duin. (0) |? 
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4 Variational and Linear Diffusion 


The relationship of neighborhood filters with classic local PDEs has been discussed 
in the previous section. Yet, the main interest has shifted to defining nonlocal 
PDEs. The extension of the neighborhood filter and the NL-means method to define 
nonlocal image-adapted differential operators and nonlocal variational methods 
starts with [47], which proposes to perform denoising and deblurring by nonlocal 
functionals. 

The general goal of this development is actually to give a variational to all 
neighborhood filters and to give a nonlocal form to the total variation as well. More 
precisely, the neighborhood filters derive from the functional 


= 2: 
su=f oe (He or) w(x — yD dxdy, 


where g and w have a Gaussian decay. In the same line, a functional yields a 
(variational) interpretation to NL-means: 


int) = [ (1 m Jax yD axa. 
2x2 


In a similar variational framework, Gilboa et al. [36] consider the general kind of 
quadratic nonlocal functional 


IW) = [ _ (ules) = uly )w (= y) ded, (6) 


where w(x, y) is any fixed weight distribution, which in most applications writes as 
the neighborhood or NL-means weight distribution. The resolution of the graph heat 
equation or the variational minimization (6) is given by 


1 
Unti(x) = ay | mame vod. 


where C(x) = fo w(x, y)dy is a normalizing factor. The freedom of having a totally 
decoupled weight distribution makes this formulation a linear and powerful tool 
for image processing. In fact, this formulation rewrites as the Dirichlet integral of 
the following nonlocal gradient: V,, u(x, y) = (u(x) — u(y))w(x, y). The whole 
process relates to a graph Laplacian where each pixel is considered as the node 
of a weighted graph, and the weights of the edge between two pixels x and y, 
respectively, are decreasing functions of the distances of patches around x and 
y, w(x, y). Then a graph Laplacian can be calculated on this graph, seen as the 
sampling of a manifold, and the linear diffusion can be interpreted as the heat 
equation on the set of blocks endowed with these weights. The eigenvalues and 
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eigenvectors of such a Laplacian can be computed and used for designing spectral 
algorithms as Wiener and thresholding methods (see [69] and [59]). 

The nonlocal term (6) has shown to be very useful as a regularization term for 
many image-processing tasks. The nonlocal differential operators permit to define 
a total variation or a Dirichlet integral. Several articles on deblurring have followed 
this variational line: [36, 44,53] for image segmentation; [8] for fluorescence 
microscopy; [81], again, for nonlocal deconvolution; and [50] for deconvolution 
and tomographic reconstruction. In [33], a paper dedicated to another notoriously 
ill-posed problem, the super-resolution, the nonlocal variational principle, is viewed 
as “an emerging powerful family of regularization techniques,’ and the paper 
“proposes to use the example-based approach as a new regularising principle in ill- 
posed image-processing problems such as image super-resolution from several low 
resolution photographs.” For all these methods, the weight distribution is computed 
in the first iteration and is maintained during the whole iteration process. 

In this section, we will concentrate on the last nonlocal functional as a linear 
diffusion process and therefore the associated graph to the image as a heat equation; 
that is, no fidelity term will be added to the functional. 


Linear Diffusion: Seed Growing 


In [37,39], a novel method was proposed for performing multi-label, semiautomated 
medical image segmentation. The Grady segmentation method is a linearized sigma 
filter applied to propagate seed regions. 

Given a small number of pixels with user-defined labels which are called 
seeds, this method computes the probability that a random walker starting at each 
unlabeled pixel will first reach one of the pre-labeled pixels. By assigning each pixel 
to the label for which the greatest probability is calculated, a high-quality image 
segmentation can be obtained. 

With each unlabeled pixel, a K-tuple vector is assigned that represents the 
probability that a random walker starting from this unlabeled pixel first reaches 
each of the K seed points. A final segmentation may be derived from these K- 
tuples by selecting for each pixel the most probable seed destination for a random 
walker. By biasing the random walker to avoid crossing sharp intensity gradients, 
a quality segmentation is obtained that respects object boundaries (including weak 
boundaries). The image (or volume) is treated as a graph with a fixed number of 
vertices and edges. Each edge is assigned real-valued weight corresponding to the 
likelihood that a random walker will cross that edge (e.g., a weight of zero means 
that the walker may not move along that edge). By a classical result the probability 
that a random walker first reaches a seed point exactly equals the solution to the heat 
equation [9] with boundary Dirichlet conditions at the locations of the seed points, 
the seed point in question being fixed to unity, while the other seeds are set to zero. 

This idea was not quite new. Region competition segmentation is an old concept 
[82]. One can also refer to an algorithm developed for machine learning by Zhu 
et al. [83], which also finds clusters based upon harmonic functions, using boundary 
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Fig. 25 (Taken from [38].) The Grady segmentation method is a linearized sigma filter applied to 
propagate seed regions. The gray curves are user-defined seed regions. A diffusion with sigma filter 
weights computed on the original image up is applied until a steady state is attained. A threshold 
gives the black curves separating the regions of initial seeds 


conditions set by a few seed points. Ref. [67] also involves weights in the image 
considered as a graph and takes seed points. The method is also directly related to 
the recent image-coloring method of Sapiro et al. by diffusion from seeds [78] (see 
also [65]). 

Thus, the Grady segmentation method is a linearized sigma filter applied to 
propagate seed regions. Figure 25 taken from [38] illustrates the process on a two- 
chamber view of a cardiac image. The gray curves are user-defined seed regions 
roughly denoting the ventricles in the image. In that case, one of the seed regions 
is put to | and the other to 0. A diffusion with sigma filter weights computed on 
the original image up is applied until a steady state is attained. This gives at each 
pixel y a value p,(y) between 0 and 1, which is interpreted as the probability for 
y to belong to the region of the first seed. In this binary case, a single threshold 
at 0.5 gives the black curves separating the regions of both seeds. Like the active 
contour method, this method is highly dependent on the initial seeds. It is, however, 
much less sensitive to noise than the snakes method [16] and permits to initialize 
fairly far from the desired contours. We will see that by the histogram concentration 
phenomenon, one can get similar or better results without any initialization. 

The very same process as illustrated allows to diffuse initial chromatic informa- 
tion on an initial gray image as we exposed in the introduction. Figure 26 illustrates 
this application and compares the obtained solution by using the NL-means and the 
neighborhood filter. 


Linear Diffusion: Histogram Concentration 
The segmentation process can be accomplished by iterating the neighborhood filter 


and computing the weight distribution in the initial image, as displayed in Fig. 27. 
The top image shows one slice of a 3D CT image with interest area surrounded 
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Fig. 26 Left and from top to bottom: initial chromatic data on the gray image, linear diffused seeds 
by using neighborhood filter weights on the gray image, and the same for the NL-means weights. 
Right: details of left-hand images. The neighborhood filter weights are not robust since just a 
single point from different objects can be easily confused and iteration may lead to an incorrect 
colorization 


by a parallelepiped. The next row shows several slices of this area of interest. It 
can be appreciated, first, that the background of arteries has a lot of oscillating 
clutter and, second, that the gray level value in arteries varies a lot, thus making 
an automatic threshold problematic. The best way actually to convince oneself 
that even in this small area a direct threshold would not do the job is to refer 
to the histograms of Fig.29. The first histogram that is Gaussian-like and poorly 
concentrated corresponds to the background. The background mode decreases 
slowly. On the far right part of the histogram, one can see a small pick corresponding 
to very white arteries. The fixing of an accurate threshold in the slowly decreasing 
background mode is problematic. The top right histogram shows what happens after 
the application of a median iterative filtering (the mean curvature motion). The 
histogram does not concentrate at all. The bottom left histogram is obtained after 
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Fig. 27 Comparative behavior of discussed methods in 3D. Application to a 3D angiography CT 
image of the head where blood vessels should be segmented. Top: one slice image of the CT 
volume data with marked interested area. Middle: display of interest area for several slices of the 
3D image. Second row: filtered slices by using median filter. Third row: sigma filter. Fourth row: 
3D nonlocal heat equation. Bottom: filtered slices by using the linear method with 3D NL-means 
weights. The whole sequence has been treated as a 3D image with a weight support of (5 x 5 x 3) 
and a comparison window of 3 x3 x3. The background is flattened and blood vessels are enhanced. 
Thus, a better segmentation is possible by a simple threshold, as justified by Fig. 2° 


applying the linearized neighborhood filter. The bottom right histogram is the one 
obtained by the linearized NL-means described in the same section. In both cases, 
one observes that the background mode of the histogram is strongly concentrated 
on a few gray level values. An automatic threshold is easily fixed by taking the first 
local minimum after the main histogram peak. This histogram concentration is very 
similar to the obtained by the mean-shift approach [20] where the neighborhood 
filter is nonlinearly iterated. In that case, the authors show that clusters tend to its 
mean, yielding piecewise constant image. 
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Fig. 28 From top to bottom and left to right: original iso-surface of the 3D image, same iso- 
surface filtered by iterative median filter, by linear sigma filter, and by linear NL-means. The iso- 
surface extracted from the original image presents many irregularities due to noise. The median 
filter makes them disappear, but makes important parts disappear and some vessels disconnect or 
fuse. Linear NL-means keeps most vessels and maintains the topology 


The histogram concentration phenomenon is actually visible in the comparative 
evolution of some slices under the various considered filters, as shown in Fig. 27. 
The first row shows these slices picked in the interest area. The topology killing 
effect of the median filter (mean curvature motion)is as follows: small arteries tend 
to vanish and larger ones shrink and become circular as shown in the third slice 
showing an artery section. The third row is dedicated to the linear sigma filter, 
which corresponds to Grady’s method applied directly to the image instead of using 
seeds. It is quite apparent that well-contrasted objects are well maintained and the 
contrast augmented, in agreement with the consistency of this recursive filter with 
the Perona-Malik equation. However, the less contrasted objects tend to vanish 
because, on them, the evolution becomes similar to an isotropic heat equation. 
The fourth row is the result of applying the 3D nonlocal linear heat equation, 
where the Laplacian coefficients are computed from the original image. The whole 
sequence has been treated as a 3D image with a weight support of (7 x 7 x 3) and 
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Fig. 29 Gray level histogram of 3D areas of interest. Top left: original 3D image before. Top right: 
after median filtering. Bottom left: after proposed method with sigma filter weights. Bottom right: 
proposed method with NL-means weights. The background is now represented by a few gray level 
values when the volume is filtered by the proposed method. A threshold can therefore be more 
easily and automatically applied 


a comparison window of 3 x 3 x 3. Clearly the background is flattened and blood 
vessels are enhanced on this background. A threshold just above the homogeneously 
made background level should give back arteries, and this indeed occurs. Thus, in 
that case, the 3D visualization of objects with complex topology like the cerebral 
arteries can be achieved by an automatic threshold as illustrated in Fig.28. The 
exact segmentation of the artery is a more difficult problem. Even if the histogram 
is concentrated, a different choice of the visualization threshold can produce slightly 
different surfaces. 


5 Conclusion 


This chapter has introduced neighborhood filters and reviewed the impact that they 
have had in many image-processing problems during these last years. 

The neighborhood filters have been analyzed under three different frameworks; 
a denoising filter, a local smoothing filter and in a variational formulation. 

As a denoising algorithm, the denoising filter has motivated state of the art 
algorithms as its generalization the NL-means algorithm. As a local smoothing filter, 
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its asymptotic analysis leads to the well known Perona-Malik equation. And its 
variational formulation is effectively used for segmentation and diffusion purposes, 
for example, in medical image analysis. 

Neighborhood filters remain as one of the main contributions in image processing 
of the last years and still influences current and future research. 
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Abstract 


Following their success in image processing (see Chapter » Local Smoothing 
Neighborhood Filters), neighborhood filters have been extended to 3D surface 
processing. This adaptation is not straightforward. It has led to several variants 
for surfaces depending on whether the surface is defined as a mesh, or as a raw 
data point set. The image gray level in the bilateral similarity measure is replaced 
by a geometric information such as the normal or the curvature. The first section 
of this chapter reviews the variants of 3D mesh bilateral filters and compares 
them to the simplest possible isotropic filter, the mean curvature motion. 

In a second part, this chapter reviews applications of the bilateral filter to 
a data composed of a sparse depth map (or of depth cues) and of the image 
on which they have been computed. Such sparse depth cues can be obtained 
by stereovision or by psychophysical techniques. The underlying assumption 
to these applications is that pixels with similar intensity around a region are 
likely to have similar depths. Therefore, when diffusing depth information 
with a bilateral filter based on locality and color similarity, the discontinuities 
in depth are assured to be consistent with the color discontinuities, which is 
generally a desirable property. In the reviewed applications, this ends up with 
the reconstruction of a dense perceptual depth map from the joint data of an 
image and of depth cues. 


1 Introduction 


The idea of processing a pixel relatively to its similar looking neighbors proved to be 
very powerful and was adapted to solve a huge variety of problems. Since its primary 
goal is to denoise data and since the same denoising problem appeared for 3- 
dimensional surfaces, the idea of a 3D bilateral filter was only natural. Nevertheless, 
we shall see that this extension is far from straightforward. Multiple adaptations 
have in fact been introduced; experimental results show that it is far better for 
denoising a shape while preserving edges than an isotropic filter (as one could 
expect). 

The bilateral filter could be used not only to filter images but also to diffuse 
information across an image: in numerous applications some information (e.g., 
depth value) is given only at some point positions. The problem is then to extrapolate 
the information for all pixels in the image. This can be used to improve the quality 
of disparity maps obtained by stereoscopy or to diffuse depth cues in images. 

In the present chapter the different applications will be reviewed and tested 
experimentally. Section 2 reviews bilateral filters applied to 3D data point sets, 
often organized in a triangulation (a mesh). It ends up with comparative simulations 
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illustrating the advantage of bilateral filters on isotropic filtering. Section 3 considers 
the various cases where, in an image, depth values or depth cues are available and 
shows that the bilateral filter used as a diffusion tool performs well in restoring a 
dense depth map. 

A previous review by Paris et al. [33] discusses the bilateral filter and its 
implementation. It also provides an overview of numerous applications. 


2 Bilateral Filters Processing Meshed 3D Surfaces 


This section proceeds by first examining the various adaptations of bilateral filtering 
on meshes (triangulated 3D surfaces) and discussing their implementation, which 
can depend on the surface triangulation. Finally several comparative experiments 
on synthetic and real meshes will be performed. Since a common notation is needed 
for all methods, this section starts with a small glossary and notation summary to 
which the reader may refer. 


Glossary and Notation 


e M: the mesh, namely, a set of triangles 

* vy: current mesh vertex to be denoised 

¢ N(v): neighborhood of vertex v (this neighborhood excludes v). 

* ny, Np, etc.: normals at vertex v or point p, etc., to the underlying surface 

* wi(||p — vI|), w2o(< ny, p —v >), etc.: 1D centered Gaussians with various 
variances, used as weighting functions applied to the distance of neighbors to 
the current vertex and to the distance along the normal direction at v. 

° H,, Hp, etc.: curvatures of the underlying surface at v, p, etc. 

¢ f: triangle of a mesh 

* ay: area of triangle f 

* cy: barycenter of triangle f 

* ny: normal to triangle f 

¢ II: projection on the plane containing triangle f 

e V: voxel containing points of the data set 

° os’, v’, p’,n’: processed versions of s, v, p, My, etc. 

* ||p —q|l: Euclidean distance between points p and qg 


The neighborhood filter or sigma filter is attributed to Lee [26] in 1983 but 
goes back to Yaroslavsky and the Sovietic image processing theory (see the book 
summarizing these works [44]) in 2D image analysis. A recent variant by Tomasi 
and Manduchi names it bilateral filter [37]. The bilateral filter denoises a pixel by 
using a weighted mean of its similar neighbors gray levels. In the original article, 
the similarity measure was the difference of pixel gray levels, yielding for a pixel v 
of an image J with neighborhood N(v): 
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where w et w2 are decreasing functions on Rt (e.g., Gaussian) and C(v) is a 
normalizing coefficient: C(v) = pent w1(|| p —v||)w2(|Z(v) —1(p)|). Thus /(v) 
is an average of pixel values for pixels that are similar in position but also in value, 
hence the “bilaterality.” 


Bilateral Filter Definitions 


Filtering without losing the sharp features is as critical for surfaces as it is for 
images, and a first adaptation of the bilateral filter to surface meshes was proposed 
by Fleishman, Drori, and Cohen-Or in [15]. Consider a meshed surface M with 
known normals n, at each vertex position v. Let \’(v) be the one-ring neighborhood 
of v (i.e., the set of vertices sharing an edge with v). Then the filtered position of v 
writes v = v + dv-n,, where 


1 
v=—— > wi(llp—vil)wi(< my, p—v >) < my, p—v > (1) 
Civ) 
pEN(v) 
where the weight normalization factor is C(v) = Ven wi(||p — vl|)wa(< 


ny, p — v >). In a nutshell, this means that the normal component of the vertex 
v is moved by a weighted average of the normal components of its neighboring 
points which are also close to the plane tangent to the surface at v. The distance 
to the tangent plane plays for meshes the role that was played for images by the 
distance between gray levels. If v belongs to a sharp edge, then the only points close 
to the tangent plane at v are the points on the edge. Thus, the edge sharpness will 
not be smoothed away. One of the drawbacks of the above filter is clearly the use of 
a mesh-dependent neighborhood. In case of a mesh with fixed length edges, using 
the one-ring neighborhood is the same as using a fixed size neighborhood. Yet in 
most cases mesh edges do not have the same length. The one-ring neighborhood is 
then very dependent on the mesh representation and not on the shape itself. This is 
easily fixed by defining an intrinsic Euclidean neighborhood. 

Another adaptation of the 2D bilateral filter to surface meshes is introduced by 
Jones, Durand, and Desbrun in [20]. This approach considers the bilateral filtering 
problem as a robust estimation problem for the vertex position. A set of surface 
predictors are linked to the mesh M: for each triangle f the position estimator IT + 
projects a point to the plane defined by f. Let ay be the surface area and cr be the 
center of f. Then, for each vertex v, the denoised vertex is 


j 1 


Y= cH Yo Te O)apwi(lle¢ — vil)w2( 1 ¢@) — vil) (2) 


fem 
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where C(v) = DEM arwi(|\c¢ — v||)w2(|| 1 ¢() — v]]) is the weight normalizing 
factor and w; and w2 are two Gaussians. 

Thus, the weight wj(||c ¢ — v||) is small if the triangle f is close to v. This term 
is the classic locality-in-space term of the bilateral. Similarly, w2(||I1 ¢(v) — vl) 
measures how far point v is from its projection onto the plane of the triangle. This 
weight favors the triangles f whose plane is coherent with v. 

Since the projection on the tangent planes operator I ¢ depends on the normals to 
jf, these normals must be robustly estimated. Normals being first-order derivatives, 
they are more subject to noise than vertex positions. Hence the method starts by 
denoising the normal field. To do so, the mesh is first smoothed using the same 
formula as above without the influence weight wz and with II ¢(v) = cy, namely, 
an updated position: 


1 
y= rare rs crarwi(ller¢ — vI)) 
feM 


where C(v) = Do rey 4fWi(|l¢¢ — v|]). The normal for each face in the denoised 
mesh is then computed and assigned to the corresponding face of the original noisy 
mesh. It is with this robust normal field that the bilateral filter of Eq. (2) is applied 
in a second step. 

The idea of filtering normals instead of point positions is crucial in point 
rendering applications, as was pointed out by Jones, Durand, and Zwicker in 
(21]. Indeed, when rendering a point set, removing noise from normal is more 
important than removing noise from point position, since normal variations are in 
fact what is perceived by observers. More precisely the eye perceives a dot product 
of the illumination and the normal, which makes it very sensitive to noisy normal 
orientations. The bilateral filter of [20] is seen as a deformation F of the points: 
v = F(v). Then, the update of normal n, can be obtained through the transposed 
inverse of the Jacobian J(v) of F(v): 


OF 
n’, = J" (v)n,, where J;(v) = a Od 
Vi 


where J; is the 7th column of J and v; is the ith component of v. 1, must then be 
renormalized. The rendering of the point set with smoothed normal is better than 
without any smoothing. 

In [38], Wang introduces a related bilateral approach which denoises feature- 
insensitive sampled meshes. Feature insensitive means that the mesh sampling 
is independent of the features of the underlying surface, e.g., uniform sampling. 
The algorithm proceeds as follows: it detects the shape geometry (namely, sharp 
regions), denoises the points, and finally optimizes the mesh by removing thin 
triangles. The bilateral filter is defined in a manner similar to [20], with the 
difference that only triangles inside a given neighborhood are used on this definition. 
Let v be a mesh vertex, N’(v) be the set of triangles within a given range of v, and 
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nf, af, cf be, respectively, the normal, area, and center of a facet f (a triangle). 
Denote by IT /(v) the projection of v onto the plane of f, and then the denoised 
vertex is defined by 


1 


where C(v) = Do rena @ewillley — vil)wo(| 1 ¢() — vi]) Qweight normalizing 
factor). 

The first step is to detect sharp regions. Several steps of bilateral filtering (as 
defined in [20]) are applied, and then a smoothness index is computed by measuring 
the infimum of angles between normals of faces adjacent to v. By thresholding this 
measurement, the sharp vertices are selected. Triangles whose three vertices are 
sharp and whose size does not increase during the bilateral iterations are marked 
as sharp. This detection done, points are restored to their original positions. Then 
the bilateral filtering formula is applied to sharp vertices only, and the geometry 
sharpness is encoded into a data collection containing normals, centers, and areas 
of filtered triangles. Points are then restored to their original position. Each sharp 
vertex is moved using the bilateral filtering over the neighboring stored data units, 
and thin vertices are removed from the mesh (these last two steps are iterated a 
certain number of times). Finally, a post-filtering step consists in applying one step 
of bilateral filtering on all non-sharp edges. 

In [40] (Wang, Yuan, and Chen), a two-step denoising method combines the 
fuzzy C-means clustering method (see Dunn’s article on fuzzy means [12]) with 
a bilateral filtering approach. Fuzzy C-means is a clustering technique that allows 
a piece of data to belong to two different clusters. Each point p gets a parameter 
/4p,k Which measures the degree of membership of p to a cluster k. Let m, be the 
number of points in the spherical neighborhood of a point p. If m, < threshold, 
the point is deleted. Otherwise, a fuzzy C-means clustering center c, is associated 
with p. The normal at point c, is computed as the normal to the regression plane 
of the data set in a spherical neighborhood of p. Fleishman’s bilateral filter [15] is 
used to filter c; which yields the denoised point. This hybrid and complex method 
is doubly bilateral. Indeed, the previous C-means clustering selects an adapted 
neighborhood for each point and replaces it by an average which is by itself the 
result of a first bilateral filter in the wide sense of neighborhood filter. Indeed, the 
used neighborhood for each point depends on the point. The second part of the 
method therefore applies a second classical bilateral method to a cloud that has 
been filtered by a first bilateral filter. 

The bilateral filtering idea was also used as a part of a surface reconstruction 
process. In [30], for example, Miropolsky and Fischer introduced a method for 
reducing position and sampling noise in point cloud data while reconstructing 
the surface. A 3D geometric bilateral filter method for edge-preserving and data 
reduction is introduced. Starting from a point cloud, the points are classified 
in an octree, whose leaf cells are called voxels. The voxel centers are filtered, 
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representative surface points are defined, and the mesh is finally reconstructed. A 
key point is that the denoising depends on the voxel decomposition. Indeed, the 
filter outputs a result for each voxel. For a voxel V, call v its centroid with normal 
ny. Let w; and u2 be two functions weighting, respectively, || p — v||, the distance 
between a point p position and the centroid location, and 6(p,v) = (np, 7,), the 
scalar product of the normal at p and the normal at the centroid. Then the output of 
the filter for voxel V is 


Yo willl — viDw2(6(p, »)) p 


~ CO) ; ) vev 
where C(v) = Piper wi(||p — v||)u2(d(p, v)). Here w, is typically a Gaussian and 
uy is an increasing function on [0, 1]. But this filter proves unable to recover sharp 
edges, so a modification is introduced: prior to any filtering for each voxel V, points 
of V are projected onto a sphere centered at the centroid v. Each mapped point is 
given a normal 71, which has direction p — v and is normalized. The geometric 
filtering is reduced to 


v= ae > u2(d(Ap,Ny)) Pp with C(v) = ee u2(5(7ip, Ny)). 


pEeV pEeV 


Although only the similarity of normals is taken into account in the above formula, 
the filter is bilateral because the average is localized in the voxel. 

In [27], Liu et al. interpreted the bilateral filter as the association to each vertex 
v of a weighted average 


1 
= wi(||p — v||)w2(||11,() — v|)) 1p) 


pEN(v) 


where C(v) = Deen wi(||p — vi))w2(||1,(@) — v||) (@ormalizing factor) and 
II,(v) is a predictor which defines a “denoised position of v due to p,” namely, the 
projection of v on the plane passing by p and having the normal n,. For example, 
the bilateral predictor used in [15] is II,(v) = v + ((p — v)-n,)ny, and in [20], the 
used predictor is II,(v) = v + ((p — v)-np)np which is the projection of v on the 
tangent plane passing by p. With this last predictor the corners are less smoothed 
out, yet there is a tangential drift due to the fact that the motion is not in the normal 
direction n, but in an averaged direction of the n, for p € N(v). Therefore a new 
predictor is introduced: 


II,(v) =v+ ia) 
Ny Np 


This predictor tends to preserve better the edges than all other bilateral filters. 
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The question of choosing automatically the parameters for the bilateral filter 
was raised by Hou, Bai, and Wang in [19]. It was proposed to choose adaptive 
parameters. The adaptive bilateral normal smoothing process starts by searching for 
the set of triangles (7;); whose barycenters are within a given distance of a center 
triangle 7. (But this keeps a distance parameter anyway.) Then the influence weight 
parameter o; is computed as the standard deviation of the distance between normals 
||2(7;) —n(T)||. The spatial weight parameter is estimated using a minimum length 
descriptor criterion (for various scales). The estimated parameters are then used to 
get the smoothed normal. This result is finally used for rebuilding the mesh using 
the smoothed normals by Ohtake, Belyaev, and Seidel’s method described in [31]. 

The bilateral filter has proved to be very efficient to denoise a mesh while 
preserving sharp features. The trilateral filter is then a natural extension which takes 
into account still more geometric information. 


Trilateral Filters 


Choudhury and Tumblin [6] propose an extension of the trilateral image filter to 
oriented meshes. It is a 2-pass filter: a first pass filters the normals and a second 
pass filters the vertex positions. Starting from an oriented mesh, a first pass denoises 
bilaterally the vertex normals using the following update: 


n= : 
C(ny) 


Yo mpwilllp — viwo(lltp — rll) 


pEN(v) 


where C(n,) = eer. wi (|p — v|l)w2(||2p — ny ||). Then, an adaptive neighbor- 
hood \V(v) is found by iteratively adding faces near v until the normals n / of face 
f differ too much from n’.. A function F measuring the similarity between normals 
is built using a given threshold R: 


F(v, f) = Lif |\n’, —ny|| < R; 0 otherwise. 


The trilateral filter for normals filters a difference between normals. Define 
na(f) = nz —n',. Then the trilaterally filtered normal n, is 


1 
ny =n, += DY) na(f)willles — vil)wo(na(f)) FO. f) 
oO) fEN(v) 


where C(v) = reno) wi(|lcr — v|)woa(f)) FO, f). Finally, the same 
trilateral filter can be applied to vertices. Call P, the plane passing through v and 
orthogonal to n’,. Call ¢+ the projection of cy onto P, and ca(f) = |lér — cyl. 
Then the trilateral filter for vertices, using the trilaterally filtered normal 1’, writes 
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> ca(f)wilic¢ — vi)wo(na (Pf) FO, f—) 


peEN(v) 


v=v+n" 


C(v) 


where C(v) = Donen Willer — VIDw2(ca PFO, f)- 

The results are similar to [20] though slightly better. They are comparable to 
the results of [15] since both methods use the distance to the tangent plane as a 
similarity between points. 


Similarity Filters 


In [41] Wang et al. proposed a trilateral filter with slightly different principles. 
A geometric intensity of each sampled point is first defined as depending on the 
neighborhood of the point 


1 
5(p) = Cp) > Wpq <Mp.d—P> 

P qeN(p) 

with 
Woq = willlg — pll)wa(l| < 2p.9 — P > l)wa(l| Hg — All) 

and 

Cy= Dd) Wom 

qgeN(p) 


This type of filter is a trilateral filter, which means that it depends on three 
variables: distance between the point p and its neighbors gq, distance along the 
normal 1, between the point p and its neighbors q, and the difference of their mean 
curvatures H, and Hy. 

At each point, a local grid is built on the local tangent plane (obtained by local 
covariance analysis), and at each point of this grid, the geometry intensity is defined 
by interpolation. Thus, neighborhoods of the same geometry are defined for each 
pair of distinct points, and the similarity can be computed as a decreasing function 
of the L? distance between these neighborhoods. 

Since the goal is to denoise one point with similar points, the algorithm proposes 
to cluster the points into various classes by the mean shift algorithm. To denoise 
a point, only points of the same class are used. This gives a denoised geometry 
intensity 5’(p) and the final denoised position p’ = p + 6’(p)np. 

More recently the NL-means (Buades, Coll, Morel [3]) method which proved 
very powerful in image denoising was adapted to meshes and point clouds by 
Yoshizawa, Belyaev, and Seidel [47]. Recall that for an image J, the NL-means 
filter computes a filtered value J(x) of pixel x as 
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1 
IQ) = a5 [ w(x, ~Q)dy, 


an adaptive average with weights 
1 2 
wry) = exp f Galle — 1) - Hy — Pat 


and C(x) = fa w(x, y)dy. 

Here G, is a Gaussian or a compactly supported function, so that it defines a 
patch. Thus, the denoised point is a mean of pixel values with weights measuring 
the local image similarity of patches around other pixels with the patch around the 
current pixel. 

Consider now the adaptation to a mesh M. Let Q,(v) = {y € M| |v—y| < 
20}. The smoothing is done by changing v at each step: v"t! = v" + k(v")n” with 
n, the normal to M at v. Let S be the surface associated to vertex y. The following 
definitions are directly adapted from the image case (a continuous formalism is 
adopted here for clarity): 


1 
k(v) = rare) [ : w(v, y)I(y)dSy 


C(v) =4 wi, y)dSy 


2 
I(y) =<nm,y-v> 


Div, y) 
Rh 


w(v, y) = exp— 


The problem is to define the similarity kernel D. Let 03 be the half radius 
of the neighborhood used to define the geometric similarity between two points, 
and 02 be the half radius of the domain where similar points are looked for, with 
03 < 0». The local tangent plane at y is parameterized by ft; and f. For all z of 
Q,,(y), the translation ¢ is defined ast = -—(< t1,z-y >.< h,z-y >) = 
—(uz, vz), where (uz,v;,w,) are the coordinates of vertex z in the local coordinate 
system (y,f1,f2,Ny). 

A local approximation F,(u, v) by radial basis functions (RBF) is built around 
each vertex v, and the similarity kernel finally yields 


Dw.y) = [ Gon (It!) |Fo(ues v2) — Hy — 8) [Pat 
03 \V 


with [(y — t) =< n,,z—v > and Go, a Gaussian kernel with variance 03. 
Thus each vertex is compared with vertices in a limited domain around it, and 
the weighted mean over all these nodes yields the denoised position. This results in 
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a better feature preserving mesh denoising method, but at the cost of a considerably 
higher computation time. 

To improve the computation time when denoising data using neighborhood 
filters, bilateral approximations were introduced by Paris and Durand, among others, 
in [32], where a signal processing interpretation of the 2D bilateral filter is given, 
yielding an efficient approximation. Another efficient method is the Gaussian k- 
d trees introduced by Adams et al. in [1]. The proposed method was designed to 
compute efficiently a class of n-dimensional filters which replace a pixel value by a 
linear combination of other pixel values. The basic idea is to consider those filters as 
nearest neighbors search in a higher-dimensional space, for example, (r, g,b, x, y) 
in case of a 2D color image and a bilateral filter. To accelerate this neighbor search, 
a Gaussian k-d tree is introduced. Consider the nonlocal means filter which has, in 
its naive implementation, a O(n? f*) complexity for n pixels and f x f patches. To 
apply Gaussian k-d tree, the position of a pixel is set to be the patch, and the value 
is set to be the color value of the pixel. A simple Principle Component Analysis 
(PCA) on patches helps to capture the dimensions that best describe the patches. 
The Gaussian k-d tree is also used to perform 3D NL-means on meshes or point 
clouds. To produce a meaningful value to describe geometry, the idea of spin images 
is used. At each point sample p, a regression plane is estimated, and the coordinates 
of the neighboring points in the local coordinate system are used to build a histogram 
of cylindrical coordinates around (p,n,) (the spin image). This gives the position 
vector. The value of p is then set to be the difference d = p’ — p between p and 
the Laplacian filtered position p’ expressed in the local coordinate system. This 
gives the input for building the Gaussian k-d tree yielding good results for mesh 
denoising. 


Summary of 3D Mesh Bilateral Filter Definitions 


The filters reviewed in this section are almost all defined for meshes. Yet, with very 
little effort almost all of them can be adapted to unstructured point clouds by simply 
redefining the neighborhoods as the set of points within a given distance from the 
center point (spherical neighborhood). Several classic variants of bilateral filters 
were examined, but their main principle is to perform an average of neighboring 
vertices pondered by the distance of these vertices to an estimated tangent plane of 
the current vertex. This distance takes the role played by the gray level in image 
bilateral filters. It can be implemented in several ways either by projecting the 
current vertex to the neighboring triangles or by projecting the neighboring vertices 
on the current triangle or by using an estimate of the normal at the current vertex 
which has been itself previously filtered. An interesting and simple possibility is 
to directly combine distance of vertices and of their normals or even distances 
of vertices, normals, and curvatures (but this requires a previous smoothing to 
get denoised normals and curvatures). Notice that position, normal, and curvature 
characterize the cloud shape in a larger neighborhood. Thus, at this point, the 
obvious generalization of bilateral filters is NL-means, which directly compares 
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point-wise the shape of the neighborhood of a vertex with the overall shape of 
the neighborhoods of others before performing an average of the most similar 
neighborhoods to deliver a filtered neighborhood. 

Sticking to the simplicity of comparisons and to the essentials of bilateral filter, 
we shall be contented in the comparative section to illustrate the gains of the bilateral 
filter with respect to a (good) implementation of its unilateral counterpart, the mean 
curvature motion, performed by the projection of each vertex on a local regression 
plane. The remainder of this section is divided as follows: section “Comparison of 
Bilateral Filter and Mean Curvature Motion Filter on Artificial Shapes” presents 
experiments and comparisons on artificial shapes, and section “Comparison of the 
Bilateral Filter and the Mean Curvature Motion Filter on Real Shapes” presents 
results on some real shapes. 


Comparison of Bilateral Filter and Mean Curvature Motion Filter on 
Artificial Shapes 


In the following experiments, the denoising of the bilateral filter as introduced in 
[15] will be compared with the mean curvature motion (MCM). Recall that [15] 
defined the update of a point as 


1 


v= cH > wi(||p — vll)w2(< my, p-—vV >) <n, p-v> 


pEN(v) 
with 


Cv) = SY) willlp—vil)wo(< my, p—v >) 
pEN(v) 


(see first section, Eq. (1) for notations). 

The mean curvature motion used here is the projection on the regression plane: 
a vertex v with normal n, and spherical neighborhood \V(v) is projected on the 
regression plane of \V(v). In [9], Digne et al. showed that this operator was an 
approximation of the mean curvature motion: 


a] 
x = Hyn,. 


The effects of bilateral denoising are first shown on some artificial shapes. A 
cube with sidelength 5 is created with added Gaussian noise with standard deviation 
0.02 (Fig. 1). Figures | and 2 show all points of the 3D cloud seen from one side of 
the cube. Obviously, the edges seem to have some width due to the noise. 

The experiments of Fig.2a—c show the denoising power of the bilateral filter 
in terms of preserving edges and should be compared with the standard mean 
curvature motion filter (Fig. 2d—f). The comparison is particularly interesting in the 
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Fig. 1 A noisy cube with 
Gaussian noise 


Table 1 Noise estimation for the sharp edge denoising 


Input Iteration 1 Iteration 2 Iteration 5 
RMSE (bilateral) 0.01 0.0031 0.0019 0.0035 
RMSE (mcm) 0.01 0.0051 0.0085 0.0164 


corner areas. The bilateral filter implies an anisotropic curvature motion leading to 
a diffusion only in smooth parts while preserving the sharp areas. Let us now see 
how those filters perform in case of a sharp edge. An estimation of the noise for 
each of the denoising methods is shown on Table |. This estimation was obtained as 
follows: an edge was created by sampling two intersecting half-planes and adding 
Gaussian noise, the obtained edge was then denoised by bilaterally filtering an mean 
curvature motion. Finally, the root-mean-square error (RMSE) to the underlying 
model is computed. Table | tends to prove that mean curvature motion, although it 
smoothes well the noisy flat parts and also smoothes away the sharpness, whereas 
the bilateral filter tends to preserve the sharp edges better. With few iterations, the 
noisy parts are smoothed out decreasing the root-mean-square error. Then, when 
iterating the operator, the sharpness tends to be smoothed, increasing the RMSE 
again. This phenomenon is of course far quicker with the mean curvature motion 
since this filter does not preserve edges at all. 


Comparison of the Bilateral Filter and the Mean Curvature Motion 
Filter on Real Shapes 


This section starts with running some experiments on the Michelangelo’s David 
point cloud. At each step an interpolating mesh was built for visualization. 

On Fig.3, denoising artifacts created by the bilateral filter can be seen. They 
appear as oscillations, for example, on David’s cheek. These artifacts can be 
explained by the fact that the bilateral filter enhances structures. Added noise 
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Fig. 2 Bilateral and MCM iterations on the cube corner. Notice how the sharpness is much better 
preserved by the bilateral filter than by the mean curvature equation. (a) 1 bilateral iteration. 
(b) 2 bilateral iterations. (c) 5 bilateral iterations. (d) 1 MCM iteration. (e) 2 MCM iterations. 
(f) 5 MCM iterations 


structures can thus be randomly enhanced by the bilateral filter. Figure 4 shows 
that some noise remains after one iteration of bilateral denoising. The bilateral filter 
is therefore iterated with the same parameters. Then, obviously, the remaining noise 
disappears at the cost of some sharpness loss (see Fig. 5). Still, the bilateral filter 
preserves sharp features much better than the mean curvature motion (Fig. 6). This 
can also be seen on a noisy simple scan of a screw nut driver (Fig.7) and on a 
fragment of the Stanford Digital Forma Urbis Romae Project (Fig. 8). 


3 Depth-Oriented Applications 


This section focuses on the applications of the bilateral filter and its generalized 
version to depth-oriented image processing tasks. The common idea to all these 
applications is to constrain the diffusion of depth information to the intensity 
similarity between pixels. The underlying assumption is that pixels with similar 
intensity around a region are likely to have similar depths. Therefore, when 
diffusing depth information based on intensity similarity, the discontinuities in 
depth are assured to be consistent with the color discontinuities. This is often a 
desirable property, as it was noticed by Gamble and Poggio [16] and Kellman and 
Shipley [22]. 
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Fig. 3 Denoising of the David’s face. (a) Original of the David. (b) Noisy David. (c) Bilateral 
denoising. (d) MCM 


Fig. 4 Denoising of the David (back). (a) Initial noisy David. (b) Bilateral denoising. (ec) MCM 


The remainder of this section is organized as follows. Section “Bilateral Fil- 
ter for Improving the Depth Map Provided by Stereo Matching Algorithms” 
reviews the applications of the bilateral filter to stereo matching algorithms, while 
section “Bilateral Filter for Enhancing the Resolution of Low-Quality Range 
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Fig. 7 Denoising of a screw nut driver scan. (a) Initial scan. (b) Bilateral denoising. (¢) MCM 


Images” describes an application to the resolution enhancement of range images. 
Section “Bilateral Filter for the Global Integration of Local Depth Information” 
reviews applications to the estimation of depth in single images. 


Bilateral Filter for Improving the Depth Map Provided by Stereo 
Matching Algorithms 


Stereo matching algorithms address the problem of recovering the depth map of a 
3D scene from two images captured from different viewpoints. This is achieved by 
finding a set of points in one image which can be identified in the other one. In fact, 
the point-to-point correspondences allow to compute the relative disparities, which 
are directly related to the distance of the scene point to the image plane. 
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Fig. 8 Denoising of fragment “31u” of Stanford Digital Forma Urbis Romae; see Koller et al. [23] 
for an explanation of the data. (a) Initial fragment with added Gaussian noise. (b) Bilateral 
denoising. (c) MCM 


The matching process is based on a similarity measure between pixels of both 
images. Due to the presence of noise and repetitive texture, these correspondences 
are extremely difficult to find without global reasoning. In addition, occluded and 
textureless regions are ambiguous. Indeed, local image matching is not enough 
to find reliable disparities in the whole image. Because of all these reasons, the 
matching process yields either low-accuracy dense disparity maps or high-accuracy 
sparse ones. 

Improvements can be obtained through filtering or interpolation, by using median 
or morphological filters, for instance. However, their ability to do so is limited. Yin 
and Cooperstock have proposed [45] a post-processing step to improve dense depth 
maps produced by any stereo matching algorithm. The proposed method consists in 
applying an iterated bilateral filter, which diffuses the depth values. This diffusion 
relies on the original image gradient instead of the one of the depth images. This 
allows to incorporate edge information into the depth map, assuring discontinuities 
in depth to be consistent with intensity discontinuities. 

The color-weighted correlation idea underlying the bilateral filter has been 
exploited by Yoon and Kweon [46] to reduce the ambiguity of the correspondence 
search problem. Classically, this problem has been addressed by area-based methods 
relying on the use of local support windows. In this approach, all pixels in a window 
are assumed to have similar depth in the scene and, therefore, similar disparities. 
Accordingly, pixels in homogeneous regions get assigned the disparities inferred 
from the disparities of neighboring pixels. 

However, when the windows are located on depth discontinuities, the same 
disparity is assigned to pixels having different depths, resulting in a foreground- 
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fattening phenomenon. This phenomenon was studied by Delon and Rougé [8]. To 
obtain accurate results, an appropriate window should be selected for each pixel 
adaptively. This problem is addressed by Yoon and Kweon [46] by weighting the 
pixels in a given window taking into account their color similarity and geometric 
proximity to the reference pixel. 

The similarity between two pixels is then measured using the support weights 
in both windows, taking into account the edge information into the disparity 
map. Experimental results show that the use of adaptive support weights produces 
accurate piecewise smooth disparity maps while preserving depth discontinuities. 

The idea of exploiting the color-weighted correlation to reduce the ambiguity 
of the correspondence problem has been implemented in a parallel architecture, 
allowing its use in real-time applications and more complex stereo systems. See 
Yang et al. [42] and Wang et al. [39] papers which achieved a good rank in the 
Middlebury benchmark (Evaluation of the Middlebury stereo website http://vision. 
middlebury.edu/stereo/) proposed by Scharstein and Szeliski [36]. 

The bilateral filter averages the pixel colors, based on both their geometric 
closeness and their photometric similarity, preferring near values in space and color 
to distant ones. Ansar, Castano, and Matthies [2], Yoon and Kweon [46], and, more 
recently, Mattoccia, Giardino, and Gambin [28] have used the bilateral filter to 
weight the correlation windows before the stereo correspondence search. On the 
other hand, Gehrig and Franke [17] have applied the bilateral filter to obtain an 
improved and smoother disparity map. 

The interpolation of disparity maps and in particular of digital elevation models 
(DEMs) has been considered in several recent works. Facciolo and Caselles [14] 
propose to interpolate unknown areas by constraining a diffusion anisotropic process 
to the geometry imposed by a reference image and coupling the process with a 
data fitting term which tries to adjust the reconstructed surface to the known data. 
More recently, Facciolo et al. [13] have proposed a new interpolation method which 
defines a geodesic neighborhood and fits an affine model at each point. The geodesic 
distance is used to find the set of points that are used to interpolate a piecewise affine 
model in the current sample. This interpolation is refined by merging the obtained 
affine patches with a Mumford-Shah-like algorithm. The a contrario methodology 
has been used in this merging procedure. In the urban context, Lafarge et al. [25] 
use a dictionary of complex building models to fit the disparity map. However, the 
applicability of such a method is less evident because of the initial delineation of 
buildings by a rectangle fitting. 

We shall illustrate the bilateral interpolation process with experiments from 
Sabater’s Ph.D. thesis [35] where the bilateral filter is used to interpolate a sparse 
disparity map. Let qg be a point in the image 7. Consider L, C J the subimage 
where the weight is learned. For each p € L, the weight due to color similarity and 
proximity is computed. 


Color similarity: the following color distance is considered 


de (ug, Up) = ((Ru(q) = Ri(p)) + (Gu(q) - Gu(p)) + (Bulg) - Bup)y) , 
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where R,,, G,, and B, are the red, green, and blue channels of u. Then the weight 
corresponding to the color similarity between p and gq is 


de (Ug, U , 
We(p,q) = exp (“te : 
1 


Proximity: The Euclidean distance between the point positions in the image plane 
is used 


d(q, p) = ((— pi) + (@— P)?)" 


where p = (p1, p2) and gq = (q1, q2). Then the weight corresponding to proximity 
is 


Meer) . 


wa (p,q) = exp (-“ 
2 


Therefore, the total associated weight between the two points g and p is 


1 1 de(ug.upY , a(q, py 
W(p.q) = = Wwelp, .q) = = exp{- a : 
(p,q) Zz" (Pp. q)wa(P. 9) Z, p( ( Pe 12 
where Z, is the normalizing factor Z, = > We(p, q)wa(p, q) . The interpolated 
pely 
disparity map j4; is computed via an iterative scheme 


11(q.k) = >) Wp. ger (p.k - 1), 


pely 


where k is the current iteration and the initialization j47(-,0) = j(-) is the sparse 
disparity to be interpolated. 

Figures 9 and 10 show the interpolated Middlebury results (100 % density). The 
experiments demonstrate that, starting from a disparity map which is very sparse 
near image boundaries, the bilateral diffusion process can recover a reasonable depth 
map. 


Bilateral Filter for Enhancing the Resolution of Low-Quality Range 
Images 


Contrary to intensity images, each pixel of a range image expresses the distance 
between a known reference frame and a visible point in the scene. Range images 
are acquired by range sensors that, when acquired at video rate, are either very 
expensive or very limited in terms of resolution. To increase the resolution of 
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Fig. 9 Tsukuba and Venus results. For each couple of images: stereo pair of images, output of 
a sparse algorithm retaining only sure points, points (in red) are the rejected correspondences, 
interpolated version of these results and ground truth 


low-quality range images acquired at video rate, Yang et al. [43] have proposed 
a post-processing step relying on an iterated bilateral filter. The filter diffuses the 
depth values of the low-quality range image, steering the diffusion by the color 
information provided by a registered high-quality camera image. 

The input low-resolution range image is up-sampled to the camera image 
resolution. Then an iterative refinement process is applied. The up-sampled range 
image Do is used as the initial depth map to build an initial 3D cost volume co. 
The 3D cost volume c;(x, y,d) associated to the current depth map D; at the ith 
iteration is given by 
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be 


Fig. 10 Teddy and Cones results. For each couple of images: stereo pair of images, output of 
a sparse algorithm retaining only sure points, points (in red) are the rejected correspondences, 
interpolated version of these results and ground truth 


ci(x,y,d) = min (uL, (d — Dj(x,y))’) (3) 


where d is the depth candidate, L is the search range controlled by constant ju, 
and D; (x, y) is the current depth estimate. To each depth candidate, d in the search 
range corresponds a single slice (disparity image) of the current cost volume. At 
each iteration, a bilateral filter is applied on each slice of the current cost volume 
c;. This allows to smooth each slice image while preserving the edges. A new cost 
volume c?¥ is therefore generated. Based on this new cost volume, a refined depth 
map D;+, is obtained by selecting for each (x, y) the lowest cost candidate d. 
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Fig. 11 Example of depth diffusion using Eq. (4). (a) Gray-level image, where BSPs and FSPs 
are marked in white and black, respectively. (b) Depth image, where points corresponding to FSPs 
are initialized with a positive value (marked in white) and the rest of the image with value zero. (c) 
and (d) Depth images after an increasing number of iterations of the DDF 


Bilateral Filter for the Global Integration of Local Depth Information 


Following the phenomenological approach of gestaltists [29], the perception of 
depth in single images results from the global integration of a set of monocular depth 
cues. However, all methods proposed in the computer vision literature to estimate 
depth in single real images rely on the use of prior experience about objects and 
their relationships to the environment. As a consequence, these methods generally 
rely on strong assumptions on the image structure [7,18], for instance, that the world 
is made of ground/horizontal planes and vertical walls, or assumptions on the image 
content [34] such as the prior knowledge of the class of objects being involved. 

In contrast to the state of the art, Dimiccoli, Morel, and Salembier [11] proposed 
a general low-level approach for estimating depth in single real images. In this 
approach the global depth interpretation is directly inferred from a set of monocular 
depth cues, without relying on any previously learned contextual information nor 
on any strong assumption on the image structure. In particular the set of initial 
local depth hypothesis derived from different monocular depth cues is used to 
initialize and constrain a diffusion process. This diffusion is based on an iterated 
neighborhood filter. 


Neighborhood Filters and the Recovery of 3D Information 1667 


With this strategy, the occlusion boundaries and the relative distances from 
the viewpoint of depicted objects are simultaneously recovered from local depth 
information, without the need of any explicit segmentation. Possible conflicting 
depth relationships are automatically solved by the diffusion process itself. 

Once monocular depth cues are detected, each region involved in a depth 
relationship is marked by one or few points, called source points (see Fig. | 1a). 
Source points marking the regions closer to the viewpoint are called Foreground 
Source Points (FSPs), whereas source points marking the regions more distant to 
the viewpoint are called Background Source Points (BSPs). In case of occlusion, 
three source points are marked (see white circle in Fig. 11a). A single FSP marks 
the region representing the occluding object, and two corresponding BSPs mark the 
partially occluded object and the background. In case of convexity, there is a single 
FSP and its corresponding BSP (see black circle in Fig. 11a). The depth image z is 
initialized by assigning a positive value A to FSPs and value 0 to BSPs. The rest of 
the image is initialized with value 0 (see Fig. 1 1b). The diffusion process is applied 
to the depth image z by using the gradient of the original image u rather than the 
one of the depth image. Doing so, the edge information is incorporated into the 
depth map, ensuring that depth discontinuities are consistent with gray-level (color) 
discontinuities. 

The depth diffusion filter (DDF) proposed in [11] by Dimiccoli, Morel, and 
Salembier is 


=lu(x)=u(y) 


1 luxy=uQ) IF 
DDFi 2) = a5 [ oe a (4) 


where S;(x) is a square of center x and side r, h is the filtering parameter which 
controls the decay of the exponential function, and 


=lu(x)—u(y) 7 
C(x) = ; e Ww dy (5) 
Sr(x) 


is the normalization factor. In practice, the parameters are r = 3 andh = 10. 

Equation (4) is applied iteratively until the stability is attained. In the discrete 
case, after each iteration, the values of FSPs and BSPs are updated. More precisely, 
if the difference between the values of an FSP and the corresponding BSP becomes 
smaller than A, then A is added to the value of the FSP. In the continuous case, 
the neighborhood filter can be seen as a partial differential equation (Buades, Coll, 
Morel [4]). With this interpretation, the depth difference constraints in the discrete 
case can be understood as the Dirichlet boundary conditions. Furthermore they 
allow to handle multiple depth layers. 

Figure 11 is an example of the diffusion through the DDF. Using Eq. (4) a very 
large number of iterations are needed to attain the stability. To make the diffusion 
faster, the following equation is used as initialization: 


1668 J. Digne et al. 


Fig. 12 Example of depth diffusion using Eq. (6) to speed up the diffusion. (a) Depth image, 
where FSPs have been initialized with a positive value (marked in gray) and the rest of the image 
with value zero. From (b) to (g) depth images corresponding to an increasing number of iterations. 
After each iteration, the depth difference between corresponding FSPs and BSPs is forced to be 
at least equal to the initial depth difference A, by adding A to FSPs when the difference between 
corresponding FSPs and BSPs becomes less than A. (h) Final depth image obtained using Eq. (4) 
on image (g) 


Slu(x)=u(y)? 
DDF,,-z(x) = sup z(yje ; (6) 
yeS;, (x) 


while Eq. (4) is used only in the last iterations (see Fig. 12). 

Experimental results on real images (see [10]) proved that this simple formula- 
tion turns out to be very effective for the integration of several monocular depth cues. 
In particular, contradictory information given by conflicting depth cues is dealt with 
the bilateral diffusion mechanism, which allows two regions to invert harmoniously 
their depths, in full agreement with the phenomenology. 
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a bi 


Fig. 13 (a) Original image. (b) Local depth cues are represented through vectors that point to the 
region closer to the viewpoint. (c) Depth image 


In Fig. 13, some experimental result involving occlusion and convexity is shown. 
For each experiment three images are shown. 

First, the original image (Fig. 13a) is shown. Then, on the second image, the 
initial depth gradient at depth cue points is represented by vectors pointing to the 
region closer to the viewpoint (red vectors arise from T-junctions, green vectors 
arise from local convexity) (Fig. 13b). Finally, the third image is the final result 
of the bilateral diffusion method (Fig. 13c). In this depth map high values indicate 
regions that are close to the camera. First and second rows of Fig. 13 show examples 
of indoor scenes, for which a proper solution is obtained. On the third row, there 
is an example of an outdoor scene involving a conflict. The T-junction detected 
on the back of the horse is due to a reflectance discontinuity, and its local depth 
interpretation is incorrect. However, on the depth map, the shape of the horse 
appears clearly on the foreground since the diffusion process allowed to overcome 
the local inconsistency. On the last row there is an example involving self-occlusion: 
occluding contours have different depth relationships at different points along its 
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Fig. 14 (/st column) Original image. (2nd column) Depth cues computed by Dimiccoli, Morel, 
and Salembier [11]: FSPs are initialized with a positive value (marked in white), and BSP and 
the rest of the images are initialized with value zero. (3rd column) Depth image computed by 
Dimiccoli, Morel, and Salembier [11]. (4th column) Local depth cues computed by Calderero and 
Caselles [5]: they are encoded through gray-level values. High values indicate pixels closer to the 
viewpoint. (5th column) Depth image computed by Calderero and Caselles [5] 


continuum. However, the bilateral diffusion method performs also well in this 
ambiguous situation. 

More recently, the idea of using a neighborhood filter to globally integrate 
local depth information in single images has been re-proposed by Calderero 
and Caselles [5]. With the aim of achieving more accurate diffusion results on 
real images, they relied on color image information to determine the adaptive 
neighborhood by using the bilateral filter of Tomasi and Manduchi [37]. In other 
words, they averaged depth information on a pixel neighborhood using color-based 
weights in the same spirit as Yang et al. [42] and Kopf et al. [24]. 

Contrary to Dimiccoli, Morel, and Salembier [11], the local depth information 
extracted by Calderero and Caselles [5] is not sparse but behaves in a continuous 
manner. More precisely, the depth value at each pixel encodes the likelihood that an 
occlusion process is taking place at that pixel (see the fourth column of Fig. 14). This 
information is computed in a multi-scale fashion and therefore has the advantage of 
being more robust with respect to the depth information extracted by Dimiccoli, 
Morel, and Salembier [11]. However, due to its local nature, their method has the 
inconvenient of not being able to handle special cases of occlusions that involve 
more global processes, such as transparency and amodal completion, as illustrated 
in Fig. 14. 


Neighborhood Filters and the Recovery of 3D Information 1671 


Due to the fact that depth information is initially estimated for each image 
pixel, Calderero and Caselles [5] use the neighborhood filter solely for assuring 
depth homogeneity of neighbor regions having similar color. Their diffusion filter is 
simpler and more robust with respect to errors coming from the estimation of local 
depth cues, but contrary to the filter of Dimiccoli, Morel, and Salembier [1 1], it does 
not allow to handle the case of multiple depth layers. 

In conclusion, the method of Calderero and Caselles [5] gives better results on 
real images, whereas the method of Dimiccoli, Morel, and Salembier [11] gives 
results that better accord to phenomenology. 


4 Conclusion 


This chapter has reviewed the use of neighborhood filters for the recovery of 3D 
information. The first part of the chapter has reviewed bilateral filters applied to 
3D data point sets, often organized in a triangulation, and has compared them to 
the simplest possible isotropic filter, the mean curvature motion, illustrating the 
advantage of bilateral filters on isotropic filtering. The second part of the chapter 
has reviewed bilateral filter applied to a data composed of a coarse, often sparse, 
depth map (or of depth cues) obtained through stereoscopy, or monocular depth 
cues estimation techniques. When diffusing depth information with a bilateral filter 
based on locality and color similarity, the discontinuities in depth are assured to be 
consistent with the color discontinuities, which is generally a desirable property. 
Experimental results have shown that the bilateral filter used as a diffusion tool 
performs well in restoring a dense depth map. 
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Abstract 


Splines and multiresolution are two independent concepts, which — considered 
together — yield a vast variety of bases for image processing and image analysis. 
The idea of a multiresolution analysis is to construct a ladder of nested spaces that 
operate as some sort of mathematical looking glass. It allows to separate coarse 
parts in a signal or in an image from the details of various sizes. Spline functions 
are piecewise or domainwise polynomials in one dimension (1D) resp. nD. 
There is a variety of spline functions that generate multiresolution analyses. The 
viewpoint in this chapter is the modeling of such spline functions in frequency 
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domain via Fourier decay to generate functions with specified smoothness in time 
domain resp. space domain. The mathematical foundations are presented and 
illustrated at the example of cardinal B-splines as generators of multiresolution 
analyses. Other spline models such as complex B-splines, polyharmonic splines, 
hexagonal splines, and others are considered. For all these spline families exist 
fast and stable multiresolution algorithms which can be elegantly implemented 
in frequency domain. The chapter closes with a look on open problems in the 
field. 


AMS Subject Classification (2010): 41A15 Spline approximation 65D07 Numer- 
ical analysis — Splines 68U10 computing methodologies and applications — Image 
processing 65T99 numerical methods in Fourier analysis 


1 Introduction 


This chapter deals with two originally independent concepts, which were recently 
combined and since together have a strong impact on signal and image analysis: the 
concept of splines, i.e., of piecewise polynomials, and the concept of multiresolution 
analysis, i.e., splitting functions — or more general data — into coarse approximations 
and details of various sizes. Ever since the combination of the two concepts, they 
have led to a load of new applications in, e.g., signal and image analysis, as well as 
in signal and image reconstruction, computer vision, numerics of partial differential 
equations, and other numerical fields. An impression of the vast area of applications 
can be gained in, e.g., [1, 17,20, 64]. 

Already, the spline functions alone proved to be very useful for mathematical 
analysis as well as for signal and image processing, analysis and representation, 
computer graphics, and many more; see, e.g., [3, 15,22, 24,35, 50,58]. An example 
for a family of spline functions are I. J. Schoenberg’s polynomial splines with 
uniform knots [59, 60]: 


1 m 
Bm) =— cye("t Veto, me N. (1) 


Here, atl denotes the one-sided power function, i.e., tt = 0 fort < Oand ath =f" 
for t > 0. The B-splines 6” can be easily generated by an iterative process. Let 
B°(t) = yW,1)(t) be the characteristic function of the interval [0, 1). Then the B- 
spline of degree m is derived by the convolution product 


p™ = p° * ee = p° xo. k p° form € N, (2) 


m+1-times 


where 
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Fig. 1 Cardinal B-splines of 
degree m = 0,..., 4 


1 
0 m—1 = 0 m—1 _ = m—1 _ 
B® B"""(x) = [ B(y) B(x — y)dy / B"-"(x— y)dy. 


For an illustration of the cardinal B-splines, see Fig. 1. 

Splines had their second breakthrough as Battle [4] and Lemarié [38] discovered 
that B-splines generate multiresolution analyses. The simple form of the B- 
splines and their compact support, in particular, were convenient for designing 
multiresolution algorithms and fast implementations. 


Definition 1. Let A : R” — R” be a linear mapping that leaves Z” invariant, i.e., 
A(Z") C Z" and that has (real or complex) eigenvalues with absolute values greater 
than 1. 

A multiresolution analysis associated with the dilation matrix A is a sequence of 
nested subspaces (V;) jez of L?(IR") such that the following conditions hold: 


G@)...C Vr CUCM C..., 
(ii) NjezV; = {0}, 
(iii) Span U;«zV; is dense in L?(R"), 
(iv) f EV) & f(A TJ EN, 
(v) fev s f(- —k) €% forallk €Z". 
(vi) There exists a so-called scaling function @ € Vo such that the family {p(e — 
k)}xez» of translates of ¢ forms a Riesz basis of Vo. 


Here, L?(IR") denotes the vector space of square-integrable functions f: R’ > C 
with norm 


f= (J. ifonleas) 


and corresponding inner product 
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(fa) = [foe at 


where g denotes the complex conjugate of g. The elements in L?(IR") are also called 
functions of finite energy. 

Riesz bases are a slightly more general concept than orthonormal bases. In fact, 
Riesz bases are equivalent to orthonormal bases and therefore can be generated by 
applying a topological isomorphism on some orthonormal basis. 


Definition 2. A sequence of functions{ f;,}nez in a Hilbert space V is called a Riesz 
sequence if there exist positive constants A and B, the Riesz bounds, such that 


Allele = |Yo, cm Ce fel], = Bllelle 


for all scalar sequences C = (cx)gezn C 1°(Z"). 
A Riesz sequence is called a Riesz basis if it additionally spans the space V. 


A good introduction to Riesz bases, their properties, and their relation to 
orthonormal bases is given in the monography by Young [75]. Multiresolution 
constructions with splines are treated in numerous sources. As a starting point, there 
are, e.g., the books by Christensen [13, 14] and Wojtaszczyk [74]. 

The mathematical properties in Definition | have intuitive interpretations. A 
function f € L(IR"), which is projected orthogonally on V;, is approximated with 
the so-called resolution A/. In fact, let 


P,: L?(R") > V; 


denote the orthogonal projection operator. Then (ii) yields that by going to lower 
resolutions, all details are lost: 


jim || Pi fl =0. 
joo 


In contrast, when the resolution is increased, 7 —> oo, more and more details are 
added. By (iii), the projection then converges to the original function /: 


lim | f —P; fl] =0. 
I7o 


Hereby, the rate of convergence depends on the regularity of f. 

The approximation spaces V; are nested, which allows for computing coarser 
approximations in V,; fork < j for functions f € V;. The scaling A* enlarges 
details. Property (iv) shows that the approximation spaces have a similar structure 
over the scales and emanate from one another. The translation invariance (v) ensures 
that the analysis of a function in V; is independent of the starting time or location. 
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Fig. 2 Left: The B-spline B° is a scaling function and operates as mean value. Right: The 
corresponding wavelet, the Haar wavelet, operates as a local difference operator 


And property (vi) finally ensures the beautiful and mighty property that the whole 
sequence of nested approximation spaces can be generated by translates and scalings 
of one single function — the scaling function. In fact, (vi) together with (iv) yields 
that 


{p(A/ e—k),k € Z"} 


is a Riesz basis for V;. 

While moving from the coarser approximation space V; to the finer, larger space 
V; +1, information has to be added. In fact, there is an orthogonal complement W;, 
J € Z, such that 


Vi41 = Vi Wj. 


These spaces are called detail spaces or wavelet spaces. It is well known that 
these spaces also possess a Riesz basis spanned by shifts of |detA|— 1 generators, 
the wavelets W1, ..., Wjaeta|-1- Here, A is the dilation matrix in Definition 1. 
The wavelets can be constructed from the scaling function. As a consequence, 
the knowledge of just the single function ¢ allows for the construction of the 
approximation spaces V; and for the wavelet spaces W;. Detailed information on 
the generation of wavelets and their properties can be found in various books, e.g., 
[16, 21,41, 44, 74]. 


Example 1. A simple example for a multiresolution analysis on L?(R) is given by 
piecewise constant functions. Consider the characteristic function @ = y(0,1) of 
the interval semi-open interval [0, 1). Then @ generates a dyadic multiresolution 
analysis, i.e., for Ad = 2. The approximation spaces are 


L*(R) 


Vj; = span {7Zj0,1)(2/ e —k)}cez 


They consist of functions constant on intervals of the form [k2~/, (k + 1)27/). The 
spaces are obviously nested and separate L7(IR) in the sense of Definition | (ii). 
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Fig. 3 Multiresolution decomposition of the step function (a) with B° as scaling function and 
with the Haar wavelet. The approximations (/eft column) are further iterated and decomposed into 
a coarser approximation and details (right column), until the coarsest approximation step, here the 
mean value, is reached. The sum of the coarsest approximation in the third iteration and of all 
details yields the original function (a). No information is lost in a multiresolution decomposition 
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Since piecewise constant functions with compact support are dense in L7(IR), (iii) 
holds. (iv) — (vi) hold by construction. In fact, this multiresolution analysis is gen- 
erated by the B-spline f° as scaling function. The B-spline basis operates as mean- 
value operator over the support interval. The corresponding wavelet extracts the 
details, i.e., the deviation from the mean value. To this end, it operates as a difference 
operator. Figure 2 shows the scaling function 8° and the corresponding wavelet, the 
so-called Haar wavelet. In Fig. 3, an example of a multiresolution is given. 


2 Historical Notes 


The idea of piecewise polynomial functions and splines goes back to Schoenberg 
[59, 60]. In the 1960s, when computer power started to be used for numerical 
procedures such as data fitting, interpolation, solving differential equations, and 
computer-aided geometric design, splines experienced an extreme upsurge. Schoen- 
berg invented and strongly contributed to the concept of cardinal splines, which have 
equidistant nodes on the integers; see, e.g., [40,61,62] and many more. 

As a parallel development, in the 1980s, the adaption of signal resolution to only 
process relevant details for a particular task evolved. For example, for computer 
vision, a multiresolution pyramid was introduced by Burt and Adelson [8]. It 
allowed to process an image first on a low-resolution level and then selectively 
increase the resolution and add more detailed information when needed. The 
definition of a dyadic multiresolution analysis, i.e, A = 2/d, was contributed by 
Mallat [43] and Meyer [46]. An interesting and in some parts historical collection 
on the most important articles in multiresolution and wavelet theory was assembled 
by Heil and Walnut [33]. 

The concepts of splines and multiresolution were joined by Lemarié [38] 
and Battle [4], when they showed that cardinal B-splines are scaling functions 
for multiresolution analyses. This led to many more developments of piecewise 
polynomial scaling functions for various settings and also multidimensions [15], 
as, e.g., polyharmonic B-splines [53, 54] and other functions inspired from radial 
basis functions [7]. 

In 1989, S. Mallat published his famous algorithm for multiresolution and 
wavelet analysis [43]. He had developed an efficient numerical method such that 
multiresolution decompositions could be calculated in a fast way. For the splines, M. 
Unser et al. proposed a fast implementation [66,68,69] which strongly contributed to 
the breakthrough of splines for signal and image analysis. In the last years, periodic, 
fractional, and complex versions of splines for multiresolution were developed, 
e.g., [11,27, 28,51, 65]. Many of them use a Fourier domain filter algorithm which 
allows for infinite impulse response filters. The former important feature of compact 
support of the cardinal B-splines and other functions is no longer a limiting criterion. 
Therefore, it can be expected that many new contributions on splines will still be 
made in the future by modeling signal and image features in Fourier domain. 


1682 B. Forster 


3 Fourier Transform, Multiresolution, Splines, and Wavelets 
Mathematical Foundations 


Regularity and Decay Under the Fourier Transform 
An important idea behind splines and multiresolution is the relation between 
regularity in time domain and decay in frequency domain and, respectively, between 
decay in time domain and regularity in frequency domain. To illustrate this, the 
notion of the Schwartz space is very useful [10, 34, 56, 63]. 


Definition 3. The subspace of functions f € C(I”) with 


sup sup (1 + [>| 2)" 1D*% f(x)| < co forall N =O, 1, 2, 


la|<N xER’ 


is called the space of rapidly decreasing functions or Schwartz space S(R”). The 
norms induce a Fréchet space topology, i.e., the space S(IR”) is complete and 
metrizable. 7 7 

Here, D® = (%) (sb) " for every multi-index w = (a,...,@,) € N'. 


The dual space S’(R”), endowed with the weak-* topology, is called space of 
tempered distributions. 


The following famous linear transform relates the viewpoints of the space domain 
and of the frequency domain: 


Definition 4. The Fourier transform, defined by 
F f(@) := fo) = / f(xet@r dx, we R’, 
R" 
is a topological isomorphism on L7(R”) and on S(IR”). Its inverse is given by 


f(x) = — | fae” “dw in L?(R") resp. in S(R"). 


(2x)" 


The Fourier transform can be extended to the space of tempered distributions. For 
T € S’(R"), the Fourier transform is defined in a weak sense as 


FT (¢): = T(¢): = T(@) forall d € S(R’). 
Also on S’(R"), the Fourier transform is a topological isomorphism. 


The Fourier transform has the nice property to relate polynomials and differential 
operators. 
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Theorem 1. 


(i) Let f € S(R). Then for allk eN 


F (f) (@) = (io) fo), 


and 


f(a) = F (ie) f) @). 


(ii) Let P be an algebraic polynomial in R", say P(x) = So, CuX{' ... xe", and 
let f € S(R"). Then 


F(p (20) 1) = Ps and Pf = PUD)f, 


where P(iD) =~, Cyil®!D®. 
(iii) Part (ii) also holds for f € S’(R"). 


Example 2. The Fourier transform of the polynomial x* is the tempered distribution 
ik 478, k No. 

For the construction of a multiresolution analysis, the scaling function can be 
used as a starting point. The idea is to choose a scaling function of a certain 
regularity, such that the generated multiresolution analysis inherits the smoothness 
properties. In particular, for the splines, the idea is to model the regularity via decay 
in Fourier domain. The following theorem gives a motivation for this. The result 
can be deduced from the considerations above, and the fact that S(IR”) is dense in 
L?(R"): 


Theorem 2. Let f ¢ L*(R") and its Fourier-transform decay as 
|f(@)| < CA+ || @ I)" 


for some ¢ > 0. Then all partial derivatives of order <N —n are continuous and in 
L?(R"). 

These results allow to construct a scaling function with explicit regularity and 
decay properties, in space and in frequency domain. However, some criteria are 
needed to verify that the constructed function generates a multiresolution analysis. 


Criteria for Riesz Sequences and Multiresolution Analyses 
The following is an explicit criterion to verify whether some function ¢ is a scaling 
function. 


1684 B. Forster 


Theorem 3. Let A be a dilation matrix and let ¢ € L?(R") be some function 
satisfying the following properties: 


(i) {b(e —k)kxezn is a Riesz sequence in L?(R"). 
(ii) ® satisfies a scaling relation. That is, there is a sequence of coefficients 
(dk )kezn such that 


G(ATx)=S°  acb(x tk) in L7(R’). (3) 


kez 
(iii) || is continuous at 0 and $(0) ¥ 0. 

Then the spaces 

V; = span {p(A! e—k)}em, j € Z, 

form a multiresolution analysis of L?(IR") with respect to the dilation matrix A. 
Proof. See, e.g., [74, Theorem 2.13] for the 1D case. 

For particular applications, the Riesz basis property (i) of {@(e—k)}xez» in Vo is 
not enough, but an orthonormal basis is needed. An example for such an application 
is the denoising of signals contaminated with Gaussian white noise [44, Chap. X, 


Sect. 10.2.1]. However, there is an elegant mathematical method to orthonormalize 
Riesz bases generated by shifts of a single function. 


Theorem 4. Let ¢ € L(R"). Then the following holds: 


(i) {b(e—k)}cegn is a Riesz sequence in L*(R") if and only if there are constants 
cand C, such that 


O<c< ee lp(w + 2mk)|? < C < 00 almost everywhere. 


That is, the autocorrelation filter M(w) := opeqn |o(w + 2mk)|? is strictly 
positive and bounded from above. 
(ii) {¢(e —k)}xezn is an orthonormal sequence if and only if 


a |p(@ + 2xk)|? = 1 almost everywhere. 


(iii) If {b(e —k)xcezn is a Riesz basis of a subspace X of L?(R"), then there exists 
a function ® € L?(R"), namely, 
. p(w) 
B(w) = id (4) 
VDeex Bo + 20k)? 
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such that {®(e — k)}xez» is an orthonormal basis of X. 


Proof. See, e.g., [74] and [44, Chap. VI]. 

Due to this theorem, every scaling function can be orthonormalized. Let @ € 
L?(R") be some scaling function that generates a multiresolution analysis {Vj} jez 
of L?(R"). Then the family {Dj xheezn with 


Dj x(x) = 24? (2-4 (x —k)), 


and @ as defined in (4) is an orthonormal basis of the space V;, j € Z. 


Example 3. A simple possibility to construct a dyadic multiresolution analysis 
in L?(R") is the tensor product approach. Let (V;)jez be a dyadic (ie, A = 
2) multiresolution analysis of L*(R) with scaling function ¢. Then (V;) jez 


with V; = V; ®---@ Vj; together with the scaling function $(x),...,%) = 
————— 
n—times 
P(x) +++ ¢(x,) forms a multiresolution analysis of L*(IR") and dilation matrix 
21d. 
In the same way, the scaling function @(x1,...,%) = @1(%1)-...+n(%) 
generates a multiresolution analysis of L*(R"), if every ¢,k = 1,...,n, isa 


scaling function of some 1D multiresolution analysis with dilation factor a € N\ 


{1}. 


Regularity of Multiresolution Analysis 

In signal and image analysis, the choice of an appropriate analysis basis is crucial. 
Here, appropriate means that the features of the basis such as smoothness should 
be in accordance with the properties of the functions to analyze. To give a blatant 
example: Analyzing a smooth signal or image with a fractal basis in general yields 
results that are difficult to interpret and to work with in practice. In this case, the 
signal resp. the image model does not match the model of the basis. 

The next section will show that the family of spline bases helps to avoid such 
difficulties, because the splines allow for a good adjustment due to their regularity 
parameter m; cf. (1) and (2). The following definition specifies the term “regular.” 
(See [74].) 


Definition 5. Denote C” the class of r-times continuously differentiable functions 
in R”,C° the class of continuous functions, and C7! the class of measurable 
functions. 

(i) Letr = —1,0,1,.... A function f : R” — C is called r-regular, iff €¢ C” and 


ae aS |< < ay a 
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for every k € No, every multi-index a with |a| < max(r, 0) and constants Ax. 
(ii) A multiresolution analysis of L?(IR") is called r-regular if it is generated by an 
r-regular scaling function. 


It is important to note that the orthonormalization procedure (4) does not affect 
the regularity of the corresponding basis. For the orthonormalized scaling function 
@ of a multiresolution analysis, the same regularity properties hold. 


Proposition 1. Let ¢ € L?(R") be anr-regular function, such that {p(e — k)}xezn 
forms a Riesz sequence. Then the via (4) orthonormalized function ® is also r- 
regular [74]. 


Order of Approximation 

Having found a scaling function that generates a multiresolution analysis, how good 
do the corresponding approximation spaces V; approximate some function f € 
L?(R") of a certain regularity? Let 


— oil 


= | +lelle)* fllzz < co}, ke No 
(27)" 


HR) ={f €V?(R"): Wf lla: 


denote the Sobolev spaces. The following criterion for the order of approximation 
turns out to be easy to verify for splines. 


Theorem 5. Let ¢ € L?(R") satisfy the following properties [23, Theorem 1.15]: 


(i) 1/¢ is bounded on some neighborhood of the origin. 
(ii) Let B, be some open ball centered at the origin and let E: = B, + (2nZ"\ 
{0}). For some a > k + n/2, all derivatives of d of order < a are in L?(E). 
(iii) DY $(a) = 0 for all |y| < k and all w € 2nZ4\ {0}. 


Then Vo = span {b(e — k)}xez» provides an approximation order k: 
For f ¢ H*(R"), 


min{|| f — s(e/h)|| 2,5 € Vo} < const. h* || f || yx for allh > 0. 


Wavelets 

For the step from a coarser approximation space V; to a finer one V; +1, information 
has to be added. It is contained in the wavelet space or detail space W;, which is the 
orthonormal complement of V; in V;+1: 


Vi41 = Vi ® W;j. 


It follows that Vi+m = Vj ® Bic) Wj +1, and hence 
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LR") = QW; (5) 
jeZ 


can be decomposed in a direct sum of mutually orthogonal detail spaces. Moreover, 
the detail spaces W; inherit the scaling property from Definition 1(iv) for the 
approximation spaces V;. For all j € Z, 


fEW) & f(A sey eM. 


The question now is whether there is also a simple basis generated by the shifts 
of one or few functions, the wavelets. The following definition is motivated from 
Eq. (5). 


Definition 6. Let A be a dilation matrix, and let {Ww }/=1 
functions in L*(IR"), such that the family 


ad €N, bea set of 


{|det A]? (Al e—k)| 1 =1,..., qgj€Zke ZY 


associated with A. 


What qualitative properties do the wavelets have? The approximation spaces V; 
are generated by the scaling function, which operates as a low-pass filter. This can be 
seen from Theorem 3 (iii) (0) # O and resp. from Theorem 5(i): 1/ b is bounded in 
some neighborhood of the origin. Therefore, the added details and thus the wavelets 
have to carry the high-frequency information. In addition, the wavelets y in Wo are 
elements of V; and therefore have the form 


WAT x) = Do POX —K) (6) 


in L? norm, where {ax}kezn are the Fourier coefficients of a certain 27 Z” -periodic 
function. 


Proposition 2. Let (V;)j¢z be a multiresolution analysis of L?(R") with respect to 


the dilation matrix A and with scaling function @. Then for a function f € L*(R") 
the following are equivalent: f € V, id and only if 


f(A"o) = m ¢(w)o(o) almost everywhere. 
Here,myf € L?((0, 27]") and 


2 1 2 
12((0,2x}") = | det A] II F ll z2qery 5 


||m + 
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For a proof, see, e.g., [21, 74]. Note that for a wavelet w as in Eq. (6), there holds 


1 : 
= ; i(w,k) 
my(o) )det A ee age : 


How many wavelets, i.e., generators of Wo, are needed to span the space? The 
parameter q in Definition 6 is yet unspecified. In fact, q depends on the scaling 
matrix. A leaves the lattice Z" invariant, AZ" C Z". The number of cosets is 
|detA| = |Z” /AZ"| (see [74, Proposition 5.5]). It turns out that q = |detA| — 1 
wavelets are needed to generate the space Wo. To motivate this, for a start, let 
ft € V, be an arbitrary function. Denote yo,...,¥q representatives of the q + 1 
cosets of AZ" in Z". Then each coset can be written as Vp, + AZ", m = 0,..., q. 
The function f has the representation 


1 


raccapet 4») =D 


ck(f)b(x — k), (7) 


kez 


or in Fourier domain 


x 1 A 
f(A’) = [act A201 PO), (8) 


in L* sense and with an appropriate 2xZ" -periodic function c¢(@) with Fourier 
coefficients (cx (f))eezn. Then c ¢(@) can be decomposed with respect to the cosets: 


_ i{a.k) __ q i(w,k) 
Cp) = Dope HPO = DD ey, page HAI 

= q i(@,Ym) i(w,k) _ q m 

~ nnn” : Deen Ch+Ym Ne a Dt ef (o), 


where 


m — pi l@.ym) i(@,k) _ ,i(@,ym) i(w,Ak) 
Cr (a) = ee ye Chetym (fe ae a CAk+ym (fe 


— i(o, m) i(A?@,k) i i(o, mn) m T 
= ” ae CAK+ Ym (fe =e : Ky (A w). 


This representation exists for all functions V,, in particular for @ and the wavelets. 
The following theorem indicates how the wavelets are constructed that generate the 
space Wo, such that Wo ® Vo = Vj. 


Theorem 6. Let @ € Vo be a scaling function and let W,...,Wq © Vi. Then the 
family {@(¢ — k)}xeze is an orthonormal system if and only if 
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ae 0 


Kg (o) = almost everywhere. (9) 


The system {o( —k)}eezn Ut = {Win (@ — k) fez is an orthonormal basis in V, 
if and only if the so-called polyphase matrix 


Ky “ aa (@) --- = (w) 
KS (c) 4, (0) + 4, (wo) 


is unitary for almost all w € R". 


The proof for a more general version of this theorem is given in [74, Sect. 5.2]. 
A summary and a condition for r-regular wavelets yields in the following 
theorem. 


Theorem 7. Consider a multiresolution analysis on R" associated with a dilation 
matrix A. 


(i) Then there exists an associated wavelet set consisting of q = |detA| — 1 
functions. 

(ii) If the multiresolution analysis is r-regular and in addition 2q + 1 > n, then 
there exists an associated wavelet set consisting of q functions, which all are 
r-regular. 


The idea of the proof is that for an r-regular function @ on R" and a 2nZ"- 
periodic C™- function n(@), the convolution defined by Wa) = = n(w)o(w) isan 
r-regular function. For an explicit proof, see again [74]. 


Example 4. As a continuation of Example 1, the wavelet function corresponding to 
= X{0,1) is derived. To this end, consider the space L?(R) and the dilation A = 2. 
Then g = detA—1 = 1; thus yo = Oand y; = | are representatives of the cosets of 
A. That is, there is only a single wavelet needed to generate Wy. Equation (7) yields 


pes aa 


1 
p(x) + Va 


5-5 


for the normalized generator of V;. Thus, co(@) = a and c;(¢) = —ke'®. This 
implies Ke () = Ky ($) = rol Then by Eq. (9) of Theorem 6, the a {d(- - 
k)}xez is orthogonal, since Ke %(@)? + Ks '(w)? = 1. The polyphase matrix 
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«9 (w) «9.(w) 

«i(a) Ki(o) 
can be completed to a unitary matrix by choosing Kp @) = -Ky, (w). The 
corresponding wavelet then has the representation 


1 1 1 
¥ (5) = yee) Fee -D. 


corresponding to (7). This yields the Haar wavelet y as illustrated in Fig. 2. 


B-Splines 


Several of the criteria for scaling functions and multiresolution analyses given in 
the previous section are based on the Fourier representation of the scaling function, 
e.g., the Riesz sequence criterion and the orthonormalization trick in Theorem 4, as 
well as the criterion for the order of approximation in Theorem 5. For this reason, 
the modeling of a scaling function in the Fourier domain to achieve certain specific 
properties is promising. 

Aiming at constructing a scaling function @ € L*(R) of regularity r = 
—1,0,1,..., this property is considered in the Fourier domain: It is a decay property 
of the Fourier transform b (compare with section “Regularity and Decay Under the 
Fourier Transform’’): 


‘ 1 
g(a) = O (Goa) for ||@|| > oo. 


Taking into account Theorem 2, a first model for the scaling function in the Fourier 
domain is 


A v(a) 
¢@) = YER, (10) 


where the function v still has to be specified. Since scaling functions satisfy a scaling 
relation (3) 


o(5)= dy oe-H in 2Q), 


the Fourier transform of this equation yields 


26(2w) = H(w)$(o), 
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An -3n -2n —-0 T 2x 3x 4n 


Fig. 4 The function |B° A | is strictly positive in the interval [—z, z] 


where (hx)xez is the sequence of Fourier coefficients of the 27-periodic function 
H. For the ansatz (10), 


_ ¢20) _ , V2) art? _ 1 ve) 
Oy d(w) — (2w)"t?2 vw) 27+! va) ’ a 


This gives the criteria for the choice of the function v: 


(i) v vanishes at the origin and there is a zero of order r + 2. This ensures that 
d € L?(R) and that Theorem 3(iii) is satisfied. 

(ii) v(2q@) is a trigonometric function, to ensure that H(w), the so-called scaling 
filter, is 27 periodic. 

(iii) v has no other zeros in [—z, x], except at the origin. Otherwise, the auto- 
correlation filter A(w) = pez lo(w + 2mk)|? would vanish somewhere, 
and the shifts of the function ¢ would fail to generate a Riesz sequence; see 
Theorem 4(i). 


A simple function ensuring all three requirements (i), (ii), and (111) is 
v(@) = (sin(@/2)0(w/2))"~, 


where @ is a 27r-periodic phase factor such that |6| = 1, i.e., a shift in time domain. 
Choosing 0(w) = e~!® yields the cardinal B-splines as given in (1) resp. (2): 


ges 1 —iw ‘ 
B°(w) _ / et dt _— 1 = = mom een 
0 1@ (0) 


Since B° has compact support, p? € C@™ [56]. Due to the convolution formula (2), 
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aa m+1 : m+1 
- on l-e‘’® sin(w/2) _,,, 
B"(@) = | ——— = ( ——__~ele/? (12) 
i@ w/2 


The 6” are scaling functions of regularity r = m — 1, as the verification of the 
criteria in Theorem 3 shows. In fact, the following holds. Let A = 2. 


Integrability: Since by (12) the functions pm are L? integrable, so are the B”", 
méN». 


Riesz sequence property: The shifted characteristic functions B°(x — k) = 
Xo.1) (x — k), k € Z, are clearly orthonormal. Theorem 4(ii) thus yields 


aa |B°(@ + 2k)|* =1 almost everywhere. 


To verify the Riesz sequence property for 6”, the autocorrelation filter must be 
bounded with strictly positive constants from above and from below. It is 


am 2 _ a0 2m+2 
vez P'@ + 27k)? = DO, [B(@ + 2k) Pt”. 


In [—2, x], |B°| is clearly positive (cf. Fig. 4), which gives |B°(x)| = 2/m asa 
positive bound from below. There is a constant c, such that 


0<c= Qi < |B°(w) |2"*? < oper |B" (w +4 2nk)\?"*?, 


Since the sequence (|B°(a + 2mk)|)cez € 1°(Z) for all w € R, the same is true 
for the sequence |B’ (w + 2k)| = |B°(@ + 2k)|"*!. This yields the existence of 


the requested upper bound c2 < oo. Thus {8 (e — k)},ez forms a Riesz sequence 
in L?(R). 


Scaling relation: The scaling filter (11) 


(1 agers _ oF ee ae m+l1 m+ 1 ok 
(1 —e-ieynt+1 au (ite) =e Dae k 7 


H(w) =2-™ 


is obviously 27-periodic and has Fourier coefficients (= oS ')) € 
keZ 


1?(Z). Hence, the B-splines satisfy the scaling relation (3) 


k=0 k 


gm (x/2) = yn Q-m ’ + ') ams + k). 


For B° this equation reads 
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Fig. 5 Scaling relation for B-splines 8”, m = 0,..., 3. The B-spline versions 8” (x/2) € V_, 
are displayed with solid lines; the scaled translates in Vo are depicted with dashed lines. The sum 
of the dashed functions gives the B-spline at the lower scale 6” (x /2) 


B°(x/2) = B(x) + BX + LD), 


which is true since B°(x/2) = x,0,1)(x/2) = x,0.2)(x). This equation and examples 
for scaling relations of other B-splines are illustrated in Fig. 5. 


Continuity and positivity of b at the origin: From Eq. (12), 
sin(w /2) |'"*! 


Broil =| 


which has a continuous continuation at the origin, and pm (0) = 1. Thus we have 
proved the following conclusion: 


Theorem 8. The cardinal B-spline B™, m € No, is a scaling function of an m — 
1-regular multiresolution analysis with dilation 2. The order of approximation is 
m+1. 
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Note that the cardinal B-splines 6” with Fourier transform of the form (12) are 
scaling functions, but they are not yet orthonormalized: The family {B”"(- —k)}cez 
spans Vo and is a Riesz basis, but it is not an orthonormal basis of Vo. Orthonormality 
can be achieved with Theorem 8 and Eq. (4): 


B”" (w) 


B"(@) := : 
Viner |B" + 2k)? 


Figure 6 shows some orthonormalized B-spline scaling functions and the corre- 
sponding wavelets. 


Polyharmonic B-Splines 


The same approach to model scaling functions in Fourier domain can be done in 
higher dimensions. We aim at constructing a scaling function for a multiresolution 
analysis of L?(IR”) of the form 


v(@) 
[| @ ||?” 


o(w) = 


réEN, r>n/2, x ER". 


With an appropriate trigonometric polynomial 


r 
’ 


v(@) = (4 ar sin? (x/2)) 


o= (@1,..-,@n), 


d is a nonseparable scaling function for a multiresolution analysis of L*(IR”) with 
respect to dilation matrices A that are scaled rotations. The corresponding function 
in space domain ¢ is called elementary r-harmonic cardinal B-spline, or short 
polyharmonic B-spline P”. This terminology can be justified as follows. The Fourier 
transform in the sense of tempered distributions of the function 1/ || @ ||” is indeed 
a polynomial — up to a logarithmic factor for 2r — n even. In fact, in S’(R"), 


FUA/ |e PY) =I x 1 (AG, 7) In |] x || +B(n,7)) =: pQ), 


with constants A(m,r), B(m,r) as given in [63, Chap. VII, §7], and A(n,r) = 0 
except for 2r —n even. (Note that for r > n/2 on the right-hand side, the Hadamard 
finite parts have to be considered.) The term polyharmonic comes from the fact that 
p is the Green function of the r-iterated Laplace operator A”. However, with these 
considerations, 


ga) =P'@)=)>0 


cage p(x +k) almost everywhere 
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m=3 


Fig. 6 Orthonormalized B-splines B” (left column) and corresponding wavelets (right column) 
for m=1, 2, 3 in time domain. Note that the orthonormalized B-splines and wavelets do not have 
compact support. Due to the orthonormalization procedure (4), the orthonormalized B-spline is an 
infinite series of shifted, compactly supported B-splines 


becomes an nD spline. Here (1%.),ez is the sequence of the Fourier coefficients of v. 
Due to the decay in Fourier domain, the polyharmonic B-spline P” has continuous 
derivatives D? for multi-indices |B| < 2r —n. In the same way as for the B-splines, 
it can be shown with the theorems given in section “Mathematical Foundations” that 
¢ forms indeed a scaling function with approximation order 2r [53,54,71]. Figure 7 
shows the polyharmonic B-spline scaling function in space domain and in frequency 
domain. 
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Fig. 7 Polyharmonic B-spline for r = 3 in space domain (/eft) and frequency domain (right) 


4 Survey on Spline Families 


There are many function families that consist of piecewise polynomials and that 
are called splines, which in addition fulfill a multiresolution condition in the one or 
the other sense. These families can be classified by various aspects, e.g., by their 
dimensionality, by the lattice which is invariant under the corresponding dilation 
matrix, by the geometries they are defined on, or whether they provide phase 
information or not, and so on. The following sections list some of these spline 
approaches and illustrates their mathematical properties and features. 


Schoenberg's B-Splines for Image Analysis: The Tensor Product 
Approach 


As mentioned in Example 3, multiresolution analyses for L*(IR") and dilation 
matrix 2/d can be generated from tensor products of 1D dyadic multiresolution 
analyses. To analyze images with B-splines, the tensor product 6” (x) 6” (y) of B- 
splines is a scaling function for L?(R?) and the dilation matrix A = 2/d. Since in 
2D the determinant det A = 4, the corresponding detail space Wo is spanned by 
three wavelet functions: 


W(x)B"(y), B'COWY), WOOVO), xyE R. (13) 


A drawback of this approach is the fact that these wavelets prefer horizontal, 
vertical, and diagonal directional features and are not sensitive to other directions; 
see Fig. 8. For the analysis of images with many isotropic features, the use of 
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Fig. 8 The three B-spline tensor wavelets (13) show a preference for horizontal, vertical, and 
diagonal directions. Here, m = 2. Minimal resp. maximal function values are given in black resp. 
white 


isotropic or steerable wavelets is recommended. However, the tensor approach is a 
simple and widely used wavelet approach. For an illustration of the respective image 
decomposition in coarse approximations and details of various sizes, see Fig. 9. 


Fractional and Complex B-Splines 


The B-splines as described up to now have a discrete order of smoothness, i.e., they 
are C” functions with n € {-1,0,1,2,...}. For some applications, e.g., in medical 
imaging, where the order of smoothness of certain image classes is fractional, it 
would be favorable to have a spline and wavelet basis that is adaptable with respect 
to this regularity [1,37, 73]. A first step in this direction was done by T. Blu and M. 
Unser, who proposed B-splines and wavelets of fractional orders [5]. They defined 
two variants of fractional B-splines, the causal ones and the symmetrical ones. 

The causal fractional B-spline is generated by applying the (aw + 1) fractional 
difference operator to the one-sided power function t+°: 


a — 1 a a _ 1 _4)k a+1 = a 
PEO = ret = peg La ("7 ) et 


The Fourier-transform representation is similar to the one of the classical B-splines 
(cf. Eq. 12): 


Here again, the smoothness property B{ € C’-’(IR) in the time domain is gained 
by the fractional polynomial decay of order O(|w|**!) in the frequency domain. 
Note that C’””(IR) denotes the Hélder space with exponent m = |a + 1] and 
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Fig. 9 Decomposition of an image [18, Part of IM008.tif] into coarse approximations and details 
of various sizes. (a) Original image. (b) Matrix of the absolute values of the multiresolution 
coefficients. Large coefficients are white. The wavelet coefficients are depicted in the lower two 
and the upper right band of each scale and the approximation coefficients in the upper left band. (c) 
Two steps of the multiresolution decomposition. From /eft to right: finest details and second finest 
details, remaining approximation of the original image. (d) Second finest details (c, center) split 


into the contribution of the three wavelets. From /eft to right: the decomposition into horizontal, 
vertical, and diagonal details 


— 


y =a+1-—M™m,iie., the space of m-times continuously differentiable functions f 
that are Hélder regular with exponent 0 < y < 1 such that there is aconstant C > 0 
with 
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|D" f(t) - D" f(s)| <C\|t—s|” VWs,te R. 


Although the fractional B-splines are not compactly supported, they decay in the 
order O(|t|~*”) as t > oo. They are elements of L'(R) for a > 0, of L?(R) 
fora > -5, and of the Sobolev spaces W, (IR) for r < a + ‘ They share many 
properties with their classical B-spline relatives, such as the convolution property, 
and their relation to difference operators, i.e., they are integral kernels for fractional 
difference operators [26] and they are scaling functions for dyadic multiresolution 
analyses. This can be verified by the procedure given in the section “B-Splines”. 

The causal fractional B-spline is not symmetric. Since for some signal and image 
analysis tasks symmetrical bases are preferred, in [5] the symmetrical fractional B- 
splines B& are proposed. They are defined in Fourier domain as follows: 


a+1 


fs) = (= ——\" (== ie = 
1@ —lW 


and therefore obviously are symmetrical in the time domain. The same regularity 
and decay properties apply as for the causal fractional B-splines. The symmetrical 
fractional B-splines are also piecewise polynomials, as long as a ¢ 2No. For even 
integer degrees, the singularity introduced through the absolute value in Eq. (14) 
causes that 62” is a sum of integer shifts of the logarithmic term |f|*” In(¢) for 
m € No. For the explicit time-domain representation and further details on these 
splines, cf. [5]. 

In [6], Blu and Unser defined another variant, the generalized fractional B-spline 
or (a, T)-fractional spline B® with a parameter t € R. Also, these splines are defined 
via their Fourier domain representation: 


F atl 4, . atl_, 
2 pie) “Ft 7 _ gio 
Br (@) = (= ) ( . ) ; 
1@ —lw 


As above, the parameter a > 0 controls the regularity of the splines. The parameter 
T, in contrast, controls the position of the splines with respect to the grid 2Z. This 
can be justified by the following fact. All variants of the B-splines considered in this 
section converge to the optimally time-frequency-localized functions in the sense of 
Heisenberg, i.e., to Gaussians or Gabor functions, if the degree a becomes large. 
For a proof for the classical cardinal B-splines, see [67]. In the case of the (a, T)- 
fractional splines [6], 


a+l 


sin(w/2) (14) 


w/2 


BI) = O (ear) for a > oo. 


This explains the notion “shift parameter” for t. Moreover, the parameter t allows to 


interpolate the spline family between the two “knots,” the symmetrical ones (t = 0) 


and the causal ones (x = att); see Fig. 10. Both parameters w and t can be tuned 
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Fig. 10 The fractional (a, t)-splines interpolate the families of the causal and the symmetric 


fractional splines. t = * 1k fork = 0, re ~ hi 1 from the most right (causal) to the most left 


2 
(symmetrical) function in each image. Right: a = 1.8. Left: a = 3 


independently and therefore allow for an individual adjustment of the analyzing 
basis. 

Another generalization are the complex B-splines [27]. There are two variants, 
both defined via their Fourier domain representation. Letz = a+iy € C,a > 
-}, y €R, and y € C. Then 


are complex B-splines of complex degree z. The functions are well defined, because 


the function 2Q(@) = si ans never touches the negative real axis such that 2(w)* 
is uniquely defined. 6* and f*, are elements of the Sobolev spaces Wy (R) for r < 


a+ 5. B* has the time-domain representation 


1 : 
Bt) = Tet) a (-1) @ ') (t—k)i, 


ie., B% is a piecewise polynomial of complex degree. For more details on the 
properties of these families of complex splines, cf. [27]. 

The idea behind the complex degree is as follows: The real part Re z = a 
operates as regularity and decay parameter in the same way as for the fractional 
B-splines. The imaginary part, however, causes an enhancement resp. damping of 
positive or negative frequencies. In fact, 


Bw) = B% (we 17 IPO) gy are 2(e) 


The imaginary part y of the complex degree introduces a phase and a scaling factor 
in the frequency domain. The frequency components on the negative and positive 
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Fig. 11 The frequency spectrum |6*“| for z = 3+ iy, y = 0,1,2 (from left to right). The 
spectrum of 6? = B+? is symmetric (right), whereas the spectra of B?*! (center) and B3*! (left) 
show an enhancement of the positive frequency axis 


real axis are enhanced with different signs, because arg (2(@) > O for negative w 
and arg §2(@) < 0 for positive w. Figure 11 illustrates this effect. 

With real-valued functions, only symmetric spectra can be analyzed. The com- 
plex B-splines, however, allow for an approximate analysis of the positive or the 
negative frequencies, because the respective symmetric bands are damped due to 
the complex exponent. However, the complex B-splines inherit many properties of 
their classical and fractional relatives. 

All of the generalized B-spline families mentioned in this section have in 
common that they are scaling functions of dyadic multiresolution analyses. They 
are one-dimensional functions, but with the tensor approach mentioned in Exam- 
ple 3 and the section “Schoenberg’s B-Splines for Image Analysis — the Tensor 
Product Approach’, they are also suitable for image processing tasks. Although the 
fractional and the complex splines, in general, do not have compact support, they 
allow for fast analysis and synthesis algorithms. Due to their closed form in Fourier 
domain, they invite for an implementation of these algorithms in Fourier domain. 


Polyharmonic B-Splines and Variants 


In the section “Polyharmonic B-Splines”, the so-called polyharmonic B-splines in 
IR” were introduced. They are defined in the Fourier domain by the representation 


P'(w) = (: pe, Sin? (@, /2) 


a WR 


, r>n/2, w= (a,...,@n). 


These polyharmonic B-splines satisfy many properties of the classical Schoenberg 
splines; e.g., they are piecewise polynomial functions, they satisfy a convolution 
relation P+’ = P" * P”, they are positive functions, etc. However, they 
do not share the property that they converge to the optimally space-frequency- 
localized Gaussians as r increases [71]. This is due to the fact that the trigonometric 
polynomial in the numerator regularizes insufficiently at the origin: The second- 
order derivative of 
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4 > %=1 sin’ (w,/2) 
kai O% 


is not continuous. Van De Ville et al. [71] therefore proposed another localizing 
trigonometric polynomial: 


n—1 


L(@) = 4 sin? (a, /2) — ; a ar sin? (wx /2) sin? (@ /2). 


k=1 
(15) 
A new function family then is defined in the Fourier domain via 
a. w)\" n 
oro) = (2S) or >t. (16) 
loll 2 


Q” is called isotropic polyharmonic B-spline. The function is piecewise polynomial 
(except for 2r—n even, where a logarithmic factor has to be added; see below) 
and shares with the polyharmonic splines their decay properties in Fourier domain 
and their regularity properties in space domain. Q” converges to a Gaussian as 
r increases, which makes the function family better suitable for image analysis 
than P”, because of the better space-frequency localization. This effect is due 
to the higher-order rotation invariance or isotropy of the localizing trigonometric 
polynomial (15): 


1 
v(o) = 1+ O(lol|’) vs. wo) = 1+ lel? + Odlel) as loll > 0. 


This causes that Or has a second-order moment, and thus the central limit theorem 
can be applied to proof the convergence to the Gaussian function. In addition, or 
has a higher regularity than Pr; therefore, Q" decays faster. For a complete proof of 
the localization property, see [71]. An example of the isotropic polyharmonic spline 
is given in Fig. 12. 

The polyharmonic B-splines and the isotropic polyharmonic B-splines both are 
real-valued functions. The isotropic B-spline is approximately rotation invariant 
and therefore is suited for image analysis of isotropic features. A complex-valued 
variant of these B-splines in 2D was introduced in [28]. The idea is to design a 
spline scaling function that is approximately rotation covariant, instead of rotation 
invariant. Rotation covariant here means that the function intertwines with rotations 
up to a phase factor. 

Again, the design of the scaling function is done in the Fourier domain, now 
using a perfectly rotation-covariant function 


1 
(wt + 03)" (@ — i@2)N’ 


Pr,N (1, @2) = 
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Fig. 12 The isotropic 
polyharmonic spline Q?. 
Compare with Fig. 7 of the 
classical polyharmonic 
B-spline P? 


where r > O and N ce N. In fact, for some rotation matrix Rg € 
GL(2,R), br.v(Rew) = e7'%®6,.y(w). For localizing the function 6,.y, the same 
trigonometric polynomials v and jz as above can be used. The corresponding 
complex polyharmonic B-splines are then defined in the frequency domain as 


N 
pr. — __@(@1.02))"" 2 

Ry 1) = regV wie? OTS a7) 
Sr.N — __ (e(@1,@2))"" 2 

Ri (1, 2) 3 (wz +3)" (w -i wy) : 


The case N = 0 yields the real-valued polyharmonic splines. 

There are also other trigonometric polynomials that are suitable as localizing 
numerators for the real and the complex polyharmonic B-splines. With an appro- 
priate choice, the features of the polyharmonic splines can be tuned [28]. However, 
for both the real and the complex variant, the localizing multiplier has to fulfill 
moderate conditions to make the respective polyharmonic B-splines become a 
scaling function. In 2D, the following result holds (cf. [28]): 


Theorem 9. Let r > O and N € No. Let n(@, 2) be a bounded, 2x Z?-periodic 
function, such that 


(n(w1, @2))"+ > 


(w? + @3)' (@1 —i@2)% 


is bounded in a neighborhood of the origin and such that n(@,, @2) # 0 for all 
(a1, @2) € [-2, t]}°\{(0, 0)}. 
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Then d = yt -p is the Fourier transform of a scaling function ¢@ which 
generates a multiresolution analysis ...V_} C Vo C Vi... of L?(R’) with dilation 


matrix A = ( 4 Peas b € Zi 
—ba 


V; = span {| det A|//2@(A/ e—k),k € Z}. 


From the Fourier domain representation immediately follows that d € L?(R?) for 
r+ x > 5 and that @ decays as |f(a1, @2)| = O(\(@1, @2) ||") when || (@, 
@2) || 00. 

As aresult, for all three variants of the polyharmonic B-splines, the classical ones 
P’ , the isotropic ones Q”, and the complex ones R’”" , the following properties hold: 
They are scaling functions for multiresolution analysis. Their smoothness parameter 
r can be chosen fractional and must fulfill 7 + x > n/2 for integrability reasons. 
Then the scaling function in space domain is an element of the Sobolov space @ € 
W; (R?) for all s < 2r + N — 1. The explicit space domain representation is 


P(x) = Do nn Mh Prin (x + Kk) 


for almost all x € R?. Here, (nx),ez2 denotes the Fourier coefficients of yt. Pr.N 
is the inverse Fourier transform of the Hadamard finite part Pf (6;.) € S’(R?). In 
fact, forr ¢ No, 


r—l 4 
Pr.n(X1,X2) = C1 (x7 + x) (x) + ix.) 


and for r € No, 
Pr.w (X1, X2) = C2 Ca + x3)" (x1 + ix2)” (in In Gig + x3) + =) 


with appropriate constants c1, C2, cz € C. This justifies the notion spline for the 
function families. They all have a closed form in the frequency domain. As in the 
case of the 1D cardinal B-splines, there is a fast analysis and synthesis algorithm 
using the frequency domain representation and the fast Fourier transform; cf. Sect. 5. 


Splines on Other Lattices 


Splines on the Quincunx Lattice 

The tensor product of two 1D dyadic multiresolution analyses yields a 2D multires- 
olution analysis with dilation matrix A = 2/d; cf. Example 3. As a consequence, the 
scaling factor while moving from one approximation space Vo to the next coarser 
space V; is |det A| = 4. For some image processing applications, especially in 
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Fig. 13 Three iterations of the quincunx subsampling scheme. (a) Z, (b) Ag 2, (c) A’ Z?. The 
thinning of the Z? lattice using the dilation matrix A, is finer than dilation with the dyadic matrix 
A = 2ld, which in one step leads from (a) to (c) 


medical imaging, this scaling step size is too large. A step size of 2 as in the 1D case 
would be preferred. Moreover, the decomposition of the wavelet space into three 
subspaces then would be avoided, and the eventual problematic of the directionality 
of the three involved wavelets would not arise. An example of a dilation matrix 
satisfying these requirement is the scaled rotation matrix 


11 
oer! 


with det A, = 2. It leads to the so-called quincunx lattice. This lattice is generated 
by applying A, to the Cartesian lattice. It holds A,f? C Z?; see Fig. 13. 

Since A, falls into the class of scaled rotations, the polyharmonic B-spline 
construction including all variants is applicable for this case; cf. Theorem 9. 
Note that the tensor product approach in general is not suitable for the quincunx 
subsampling scheme. 


Splines on the Hexagonal Lattice 

Images as 2D objects are normally processed on the Cartesian lattice, i.e., the image 
pixels are arranged on a rectangular grid. For image processing, this arrangement 
has the drawback that not all neighbors of a pixel have the same relation: The 
centers of the diagonal neighbors have a larger distance to the center pixel than the 
adjacent ones. A higher degree of symmetry has the hexagonal lattice. It is therefore 
ideal for isotropic feature representation. The hexagonal lattice gives an optimal 
tessellation of R? in the sense of the classical honeycomb conjecture, which says 
that any partition of the plane into regions of equal area has a perimeter at least 
that of regular hexagonal tiling [32]. The better isotropy of the hexagonal lattice is 
attractive for image analysis and has led to a series of articles on image processing 
methods (e.g., [31,45,52]) as well as on applications (e.g., [30, 36, 47, 72]). 

The hexagonal lattice is generated by applying the matrix 
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Fig. 14 Three iterations of subsampling of the hexagonal grid with A = Ry, BR; |, where B = 
(2, 1). (@) An, (b) AAn = Ry BR; | Ay = Ry BZ?, (c) A?Ap 


R, = 2 1 1/2 
"V7 \0 V3/2 
on the Cartesian lattice A, = R,Z*. A scaling function of a multiresolution 


analysis of L7(IR?) in the hexagonal lattice fulfills all properties of Definition 1, 
but the last two. They change to 


(iv) f ENS f(- — Rak) € Yforallk € Z?. 
(v) There exists a scaling function @ € Vo, such that the family {o(e — Rik)},er2 
of translates of forms a Riesz basis of Vo. 


Let A be a dilation matrix which leaves the hexagonal lattice invariant AA; C Ap. 
Then A is of the form [19] 


A= R,BR;' 


with B € GL(2,R) having only integer entries and with eigenvalues strictly larger 
than one. Figure 14 gives an example of two subsampling steps on the hexagonal 
lattice. 

There are several possible approaches to define spline functions on the hexagonal 
lattice. Sablonniére and Sbibih [57] proposed to convolve piecewise linear pyramids 
to generate higher-order B-splines. Van De Ville et al. [70] started with the 
characteristic function of one hexagon and also used an iterative convolution 
procedure to construct B-splines of higher degree. However, both approaches lead 
to discrete-order hexagonal B-splines. If A is a scaled rotation, then fractional and 
complex B-splines on the hexagonal lattice can be defined in an analog way as in 
the section “Polyharmonic B-Splines and Variants” for polyharmonic splines and 
their (complex) variants [19]. Consider again the perfectly rotation-covariant (or for 
N = Orotation-invariant) function 
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1 


(w? + w3)! (@1 — i@2)" 


Pr.N (@ ,@2) = 


where r > Oand N € No. The idea is now to use a hexagonal-periodic trigonometric 
polynomial for localizing this function and to eliminate the singularity at the origin. 
Condat et al. [19] proposed 


nn(0o1, 02) = (6 ~ 2 (cos (3% (oy /V3 + on)/-V9) 
+ cos(3!/4 (w,/V3 + @2)/V2) + cos (37!/44/2@))), 


and defined the elementary polyharmonic hexagonal rotation-covariant B-spline via 
its frequency domain representation as 


(nn(@1. @2))"F 


RIN (w1, 02) = 7 ; : 
: (wy + 3) (@, —i@,)% 


The B-spline in space domain then has the representation 
rN 
RY) = Do ey Mik Prin (X — Rk). 


Here, (71,4.)~e72 denotes the sequence of Fourier coefficients of ae, Figure 15 
shows the localizing trigonometric polynomial 7 and the Fourier spectra of two 
hexagonal splines. 

For N = 0, the functions are the elementary polyharmonic hexagonal rotation- 
invariant B-splines. They are real-valued functions that converge to a Gaussian, 
as r — oo, and therefore are well localized in the space domain as well as in 
the frequency domain. For N ¢€ N, the splines are complex-valued functions and 
approximately rotation covariant in a neighborhood of the origin: 


REN (w) = eiN ae) (1 +C lal? + O(loll*)) for w > 0, 


where w = (@1, @2) and C € R is a constant. 
The translates of the complex B-spline mo form a Riesz basis of the approxi- 
mation spaces 


L?(R2) 
V; = span {RAN (Aix —Rrik)ik e€ z\ ,J EZ. 


The ladder of spaces (V;) jez generates a multiresolution analysis of L?(R’) for the 
hexagonal grid and for scaled rotations A. Also in this case, the implementation of 
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Fig. 15 Localization of p” with (a) the hexagonal-periodic trigonometric polynomial nj, yields 
the elementary polyharmonic hexagonal rotation-covariant B-spline. Frequency spectrum of 
Rir, N (or equivalently of Rj\r + N/2, 0), (b) for r + * = 1, and (c) forr + 7 = 2.5 


the analysis and synthesis algorithm can be elegantly performed in the frequency 
domain [19,71]. 


5 Numerical Implementation 


For the illustration of the numerical method, we focus on the quincunx dilation 


matrix 
11 
A= 1 
& ') (18) 


and consider the polyharmonic spline variants in 2D as defined in the section 
“Polyharmonic B-Splines and Variants”. Since det A = 2, the generators of the 
multiresolution space and the corresponding wavelet space are two functions: the 
scaling function (here the variant of the polyharmonic spline) and the associated 
wavelet. We consider the scaling function ¢(x) = Q/,(x) resp. $(x) = ne (x). It 
spans the ladder of nested approximation spaces {V;} jez via 


Splines and Multiresolution Analysis 1709 


; - DR), 
V; = span {| det A|//2(A/ e —k),k € Z7} ,j €Z. 


Denote 
M(o) = Yo a P(@ + 2k)! (19) 


the autocorrelation filter. It is bounded 0 < M(w) < C for some positive constant 
C [28]. The scaling functions can be orthonormalized, applying the procedure given 
in Theorem 4(iii): 


ee o(w) . 
Vy M(w) 


The scaled shifts of ® span the same spaces {Vj} jez. 
The B-splines as scaling functions ¢@ satisfy a refinement relation 


¢(A !x) = pe heo(x —k) almost everywhere and in L7(R’). 


This relation in fact is a discrete convolution. The Fourier transform yields a 27 Z?- 
periodic function H € L*(T?) of the form 


iw) — are) 2) = brn (AT @)C(ATo) 
H(e'®) = | det A|- = |det A|- Fn voto) 


_— o(AT@) | 
i) ‘GEDA we€eR 


Here, €(@) = (4 (w))'t 2 is the localizing multiplier for the isotropic polyharmonic 
B-spline in the case N = 0 and for the rotation-covariant polyharmonic B-splines 
in the case N € N; cf. (15)-(17). 

The wavelet function w spanning a Riesz basis for the orthogonal complement 


L?(R’) 


W; = span {2//2y(A/ e—k),k € Z?} 


in V;41 = W; ® V; can also be gained in the frequency domain. For the quincunx 
dilation matrix A as in (18), a wavelet (or sometimes called a prewavelet, since the 
functions are not yet orthonormalized) is given by 


U(o) = Gle'’)o(@) = e"' Hw + (2, 2)")M(o + (a, 2)" )6(0), 


compare with the section “Wavelets”. The (pre)wavelet Riesz basis for W; is then 
given by the family {Wj4 = 2//°W(A/ e—k),k € 27}. 

This basis in general is not orthonormal: (Wjx%,Wj7) A 6x7. A function f € 
L?(R*) can then be represented by the series 
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f= weber (A Wik) Wie = DS age (A Wir) Wie: 


where (Wither denotes the dual basis for each j € Z: (Was Wji) = dx,. Its 
generator in the frequency domain is 


M(o + (x, 2)") $(@) 
M(ATow) = M(o)’ 


(wo) = eH + Gen)) 


In contrast, the formula 


i M ,m)t) » 
W(w) = | ie) 


generates an orthonormal wavelet basis. It corresponds to the orthonormal basis 
of Vo generated by the integer shifts of the orthonormalized scaling function 
@®. These considerations show that there are three variants of a multiresolution 
implementation: an “orthonormal” one with respect to the orthonormalized scaling 
functions and corresponding orthonormal wavelets; one with the B-splines on the 
analysis side, 


f= Viren (Fi bi + Do co (Biel bia for fe Vir, 


and finally one with the B-splines on the synthesis side: 
f= aa (A Pj RIP IR + ee (AV a)Wix for fe Vj4i. 
Both, the scaling filters H(e'”) and the wavelet filters 
G(e!®) = e '' A(w + (2, 2)" )M(w + (x, 2)") 


as well as their orthogonal and dual variants in our case are nonseparable and 
infinitely supported. Therefore, a spatial implementation of the decomposition 
would lead to truncation errors due to the necessary restriction to a finite number 
of samples. However, because of the closed form of H and therefore of G, the cor- 
responding multiresolution decomposition or wavelet transform can be efficiently 
implemented in the frequency domain. The respective image first undergoes an FFT 
and then is filtered in the frequency domain by multiplication with the scaling filter 
H and the wavelet filter G. This method automatically imposes periodic boundary 
conditions. 

The coefficients resulting from the high-pass filtering with G are the detail 
coefficients. They are stored, whereas the coefficients resulting from the low-pass 
filtering H are reconsidered for the next iteration step. 
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open Abra bi tie = Dyce Abia ORT DO, oF Via) Wie: 


For the details and tricks of the frequency domain implementation, cf. [25, 48, 49, 
71). 

Figure 16 shows the multiresolution decomposition for the scaling function @ = 
icone There, it was assumed that the image is bandlimited and projected on the 
space Vo, which has the advantage that the coefficients do not depend on the chosen 
flavor of the scaling function, i.e., orthogonal, B-spline, or dual. Qualitatively, the 
transform is very similar to a multiscale gradient with the real part corresponding to 
the x2 derivative and the imaginary part corresponding to the x, derivative [28]. 


6 Open Questions 


In this chapter, a method for the construction of spline multiresolution bases 
was described. It yields a nice variety of new bases with several parameters for 
adaption and tuning. In the last decade, the notion of compressive sampling or 
compressed sensing arose, which is footing on the existence of well-adaptable bases. 
In fact, the idea behind compressed sensing is that certain functions have a sparse 
representation, if the underlying basis is smartly chosen. In this case, the function 
can be reconstructed from very few samples because of the prior knowledge of 
sparsity in this underlying basis. As a consequence, the knowledge on sparsity 
allows to sample such a signal at a rate significantly under the Nyquist rate. (The 
Shannon-Nyquist sampling theorem says that a signal must be sampled at least two 
times faster than the signal’s bandwidth to avoid loss of information.) 

In the last 40 years, and virtually explosively in the last 10 years, many important 
theoretical results were proven in this field, in particular by D. Donoho, E. Candés, 
J. Romberg, and T. Tao. For an introduction and references on compressed sensing, 
see, e.g., [2,9] and the website [55]. 

Compressed sensing is based upon two fundamental concepts: that of incoher- 
ence and that of sparsity. Let {x;};=1,..~ be an orthonormal basis of the vector 
space V. Let f = ee s;x; with s; = (f,x;). The signal f is called k-sparse, if 
only k of the coefficients are nonzero, k € N. 

A general linear measurement process for signals consists in computing M < N 
inner products y; = (f, y;) for a collection of vectors {y;};. In matrix form, 


g=Yf=YXs, 


where Y and X are the matrixes with {y;}; and {x;}, as columns, and YX is an M x 
N matrix. If the function families Y and X are incoherent, 1.e., if the incoherence 
measure 


WY, X)= VN max |(vi.x/)| € [1 VN] 
1<i,j<N 
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Fig. 16 Decomposition of an image [18, Part of IM115.tif] into approximation and wavelet coef- 
ficients. (a) Original image. (b) Matrix of the absolute values of the multiresolution coefficients. 
Large coefficients are white. The approximation coefficients are in the upper left band; the other 
bands are wavelet coefficients on six scales. (c) Real part of the coefficient matrix and (d) imaginary 
part of the decomposition matrix for ¢ = R,,? and the corresponding wavelets. The coefficients 
had their intensity rescaled for better contrast 


is close to one, then under mild additional conditions, the k-sparse signal f can be 
reconstructed from M > Cy?(Y, X)k In N samples with overwhelming probability. 
Wavelet bases have proven to be very suitable for compressed sensing. It is an 
open question to classify the signals from certain applications and to estimate in 
which appropriate B-spline basis they have a k-sparse representation. Then adequate 
bases and function families incoherent with the spline bases have to be identified. 
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In the last years, the concept of sparsity entered image processing. It has proven 
to help immensely accelerate the solution of inverse problems and reconstruction 
algorithms, e.g., in medical imaging, such as in magnetic resonance imaging [42], 
computed tomography [12], photo-acoustic tomography [39], tomosynthesis [29], 
and others. In this area, as well as in other fields of imaging, it can be expected 
that the combination of splines — due to their easy modeling and the fast frequency 
domain algorithms — multiresolution and wavelets, and sparsity will lead to novel 
impressing fast algorithms for image reconstruction. 


7 Conclusion 


In the design procedure for scaling functions of multiresolution analyses, regularity 
and decay features, as well as symmetry properties, can be tuned by an appropriate 
modeling in frequency domain. The idea is to start in the frequency domain with a 
polynomial function P that fulfills the required symmetry features and that has a 
degree, such that 1/P decays sufficiently fast. This assures that the resulting scaling 
function has the desired regularity. However, 1/P in general is not an L* function 
and has to be multiplied with a localizing trigonometric polynomial v that eliminates 
the zeros in the denominator such that = becomes square integrable. The choice 
of this trigonometric polynomial has to be taken carefully to be compatible with 
the required features modeled in 1/P. Then under mild additional conditions, the 
fraction 


es 
T= 


is the scaling function of a multiresolution analysis. This construction can be 
performed for 1D and higher dimensional spaces likewise. In the time domain, 
the resulting scaling function is a piecewise polynomial, thus a spline. This design 
procedure for scaling functions unites the concepts of splines and of multiresolution. 

Interestingly, the polynomial in the denominator can be of a fractional or 
a complex degree and therefore allows a fine tuning of the scaling function’s 
properties. However, the scaling function then becomes an infinite series of shifted 
(truncated) polynomials. The numerical calculation with the approximating basis of 
the multiresolution analysis in the time domain would cause truncation errors, which 
is unfavorable. But due to the construction of ¢ in the frequency domain and due 
to the closed form there, the implementation in the frequency domain with periodic 
boundary conditions yields a fast and stable multiresolution algorithm suitable for 
image analysis tasks. 
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1 Introduction 


In contrast to classical Fourier analysis, time-frequency analysis is concerned 
with localized Fourier transforms. Gabor analysis is an important branch of time— 
frequency analysis. Although significantly different, it shares with the wavelet 
transform methods the ability to describe the smoothness of a given function in 
a location-dependent way. 

The main tool is the sliding window Fourier transform or short-time Fourier 
transform (STFT) in the context of audio signals. It describes the correlation of a 
signal with the time-frequency shifted copies of a fixed function (or window or 
atom). Thus, it characterizes a function by its transform over phase space, which 
is the time-frequency plane (TF-plane) in a musical context or the location—wave- 
number domain in the context of image processing. 

Since the transition from the signal domain to the phase space domain introduces 
an enormous amount of data redundancy, suitable subsampling of the continuous 
transform allows for complete recovery of the signal from the sampled STFT. 
The knowledge about appropriate choices of windows and sampling lattices has 
increased significantly during the last three decades. Since the suggestion goes back 
to the idea of D. Gabor [45], this branch of TF analysis is called Gabor analysis. 
Gabor expansions are not only of interest due to their very natural interpretation 
but also algorithmically convenient due to a good understanding of algebraic and 
analytic properties of Gabor families. 

In this chapter, we describe some of the generalities relevant for an understanding 
of Gabor analysis of functions on R?. We pay special attention to the case d = 
2, which is the most important case for image processing and image analysis 
applications. 

The chapter is organized as follows. Section 2 presents central tools from 
functional analysis in Hilbert spaces, e.g., the pseudo-inverse of a bounded operator 
and the central facts from frame theory. In Sect. 3, we introduce several operators 
that play important roles in Gabor analysis. Gabor frames on L?(IR“) are introduced 
in Sect. 4, and their discrete counterpart are treated in Sect. 5. Finally, the application 
of Gabor expansions to image representation is considered in Sect. 6. 


2 Tools from Functional Analysis 


In this section, we recall basic facts from functional analysis. Unless another 
reference is given, a proof can be found in [17]. In the entire section, #1 denotes 
a separable Hilbert space with inner product (-, -). 
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The Pseudo-inverse Operator 


It is well known that an arbitrary matrix has a pseudo-inverse, which can be used 
to find the minimal-norm least squares solution of a linear system. In the case of an 
operator on infinite-dimensional Hilbert spaces, one has to restrict the attention to 
linear operators with closed range in order to obtain a pseudo-inverse. Observe that 
a bounded operator (We will always assume linearity!) U on a Hilbert space 1 is 
invertible if and only if it is injective and surjective, while injectivity combined with 
a dense range is not sufficient in the infinite-dimensional case. However, if the range 
of U is closed, there exists a “right-inverse operator” U* in the following sense: 


Lemma 1. Let H, K be Hilbert spaces, and suppose that U : K + His a bounded 
operator with closed range Rv. Then there exists a bounded operator Ut :H > K 


for which 
UUtx =x, Vx € Ry. (1) 


Proof. Consider the operator obtained by taking the restriction of U to the 
orthogonal complement of the kernel of U, i.e., let 


U := Ung Na > H. 


Obviously, U is linear and bounded. U is also injective: if Ux = 0, it follows that 
xe Nee Ny = {0}. We prove next that the range of U equals the range of U. 
Given y € Ry, there exists x € K such that Ux = y. By writing x = x1 + x2, 
where x; € N+, x2 € Ny, we obtain that 


Ux = Ux, = U(x, +») = Ux= y. 
It follows from Banach’s theorem that U has a bounded inverse 
‘tie ; Ru > Ne 


Extending os! by zero on the orthogonal complement of Ry, we obtain a bounded 
operator Ut : H — K for which UU'x = x forallx € Ry . Bi 


The operator U? constructed in the proof of Lemma 1 is called the pseudo-inverse 


of U. In the literature, one will often see the pseudo-inverse of an operator U defined 
as the unique operator U? satisfying that 


Nat = Res Real: and UUtx =x,x € Ry: (2) 


this definition is equivalent to the above construction. We collect some properties of 
U? and its relationship to U. 
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Lemma 2. Let U : K — H be a bounded operator with closed range. Then the 
following holds: 


(i) The orthogonal projection of H. onto Ry is given by UU'. 
(ii) The orthogonal projection of K onto Ry: is given by UU. 
(iii) U* has closed range, and (U*)' = (U")*. 
(iv) On Ruy, the operator U* is given explicitly by 


Ut =U*(UU*). (3) 


Bessel Sequences in Hilbert Spaces 


When we deal with infinite-dimensional vector spaces, we need to consider 
expansions in terms of infinite series. The purpose of this section is to introduce 
a condition that ensures that the relevant infinite series actually converge. When 
speaking about a sequence { fx}? in H, we mean an ordered set, i.e., { fc }P, = 
{fi, fa,...}. That we have chosen to index the sequence by the natural numbers is 
just for convenience. 


Definition 1. A sequence { f;,}?°_, in H is called a Bessel sequence if there exists a 
constant B > 0 such that 


DMA A? < BISI?, VF eH. (4) 
k=1 


Any number B satisfying (4) is called a Bessel bound for { fi, }?2.,. The optimal 
bound for a given Bessel sequence { f,}?2., is the smallest possible value of B > 0 
satisfying (4). Except for the case f; = 0, Vk € N, the optimal bound always 
exists. 


Theorem 1. Let { fi, }?2., be a sequence in H and B > 0 be given. Then { fx}? , is 
a Bessel sequence with Bessel bound B if and only if 
[oe 
T : {cg}pe > Yo ck fi 
k=1 


defines a bounded operator from €?(N) into Hand ||T|| < VB. 


The operator T is called the synthesis operator. The adjoint T* is called the 
analysis operator and is given by 


T*:HOPN), Tf =U ARG 
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These operators play key roles in the theory of frames, to be considered in 
section “Frames and Their Properties.” 

The Bessel condition (4) remains the same, regardless of how the elements 
{ fc}? are numbered. This leads to a very important consequence of Theorem |: 


Corollary 1. If { f,}?2, is a Bessel sequence in H, then \~?-; cx fi converges 
unconditionally for all {cx}P-_) € £?(N), ie., the series is convergent, irrespective 
of how and in which order the summation is realized. 


Thus, a reordering of the elements in {/,}?2, will not affect the series 
ye ck fe When {c,}P2, is reordered the same way: the series will converge 
toward the same element as before. For this reason, we can choose an arbitrary 
indexing of the elements in the Bessel sequence; in particular, it is not a restriction 
that we present all results with the natural numbers as index set. As we will see in 
the sequel, all orthonormal bases and frames are Bessel sequences. 


General Bases and Orthonormal Bases 


We will now briefly consider bases in Hilbert spaces. In particular, we will discuss 
orthonormal bases, which are the infinite-dimensional counterparts of the canonical 
bases in C”. Orthonormal bases are widely used in mathematics as well as physics, 
signal processing, and many other areas where one needs to represent functions in 
terms of “elementary building blocks.” 


Definition 2. Consider a sequence {e, }?2 , of vectors in H. 


(i) The sequence {e;,}?2, is a (Schauder) basis for H if for each f € H, there 
exist unique scalar coefficients {cx (f)}?2, such that 


f= ae (5) 
k=1 


(ii) A basis {e,}?2., is an unconditional basis if the series (5) converges uncondi- 
tionally for each f € H. 

(iii) A basis {e,}?2, is an orthonormal basis if {e,}?2, is an orthonormal system, 
ie., if 


1 if k=j, 


(€x,€j) = bk j = 0 if k#{. 


An orthonormal basis leads to an expansion of the type (5) with an explicit 
expression for the coefficients c; (f): 
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Theorem 2. [f {e,}72, is an orthonormal basis, then each f € H has an 
unconditionally convergent expansion 


f= Dilherder- (6) 


In practice, orthonormal bases are certainly the most convenient bases to use: for 
other types of bases, the representation (6) has to be replaced by a more complicated 
expression. Unfortunately, the conditions for {e,}?2, being an orthonormal basis 
are strong, and often it is impossible to construct orthonormal bases satisfying extra 
conditions. We discuss this in more detail later. Note also that it is not always a 
good idea to use the Gram—Schmidt orthonormalization procedure to construct an 
orthonormal basis from a given basis: it might destroy special properties of the basis 
at hand. For example, the special structure of a Gabor basis (to be discussed later) 
will be lost. 


Frames and Their Properties 


We are now ready to introduce one of the central subjects: 


Definition 3. A sequence { f,}?2., of elements in H is a frame for H if there exist 
constants A, B > 0 such that 


APS ie Se Bel. Ve et (7) 
k=1 


The numbers A and B are called frame bounds. A special role is played by frames 
for which the optimal frame bounds coincide: 


Definition 4. A sequence { /,}?2, in H is a tight frame if there exists a number 
A > 0 such that 


KA fe)? = ANFIP, VE eH. 


Me 


> 
ll 
- 


The number A is called the frame bound. 


Since a frame { f,}?2, is a Bessel sequence, the operator 


T: OW) > H, Tladpi =) ce te (8) 
k=1 
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is bounded by Theorem 1. Composing T and T*, we obtain the frame operator 


S:HOUH, Sf=TI* f= dU fe fe (9) 


k=1 


The frame decomposition, stated in (10) below, is the most important frame 
result. It shows that if {f,}?2, is a frame for H, then every element in H has a 
representation as an infinite linear combination of the frame elements. Thus, it is 
natural to view a frame as a “generalized basis.” 


Theorem 3. Let { f,}?2., be a frame with frame operator S. Then 


f= URS fe) fer VIE, (10) 
k=1 
and 
f= VASA. VEE. (11) 
k=1 


Both series converge unconditionally for all f € H. 


Theorem 3 shows that all information about a given vector f € H is contained 
in the sequence {( f, S~' f,)}22,. The numbers (f, S~! f;) are called frame coef- 
ficients. The sequence {S~! fi,}2, is also a frame; it is called the canonical dual 
frame of { fi }R2). 

Theorem 3 also immediately reveals one of the main difficulties in frame theory. 
In fact, in order for the expansions (10) and (11) to be applicable in practice, we need 
to be able to find the operator S~! or at least to calculate its action on all f,, k € N. 
In general, this is a major problem. One way of circumventing the problem is to 
consider only tight frames: 


Corollary 2. If { fi }?2, is a tight frame with frame bound A, then the canonical 
dual frame is {A7' fi}°2,, and 


f= a: te) tes VA EH. (12) 


k=1 


By a suitable scaling of the vectors { f,}?2, in a tight frame, we can always obtain 
that A = 1; in that case, (12) has exactly the same form as the representation via an 
orthonormal basis; see (6). Thus, such frames can be used without any additional 
computational effort compared with the use of orthonormal bases; however, the 
family does not have to be linearly independent now. 
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Tight frames have other advantages. For the design of frames with prescribed 
properties, it is essential to control the behavior of the canonical dual frame, but 
the complicated structure of the frame operator and its inverse makes this difficult. 
If, e.g., we consider a frame { f,}?2, for L?(R) consisting of functions with 
exponential decay, nothing guarantees that the functions in the canonical dual frame 
{S| f, }P2., have exponential decay. However, for tight frames, questions of this 
type trivially have satisfactory answers, because the dual frame equals the original 
one. Also, for a tight frame, the canonical dual frame automatically has the same 
structure as the frame itself: if the frame has Gabor structure (to be described in 
Sect. 4), the same is the case for the canonical dual frame. 

There is another way to avoid the problem of inverting the frame operator S. 
A frame that is not a basis is said to be overcomplete; in the literature, the term 
redundant frame is also used. For frames { f,}?2 , that are not bases, one can replace 
the canonical dual {S~' fi, }°2., by other frames: 


Theorem 4. Assume that { f,}?2, is an overcomplete frame. Then there exist 
frames {gx }e—) Z {S7! fic}, for which 


f=DVihedh. When. (13) 


k= 


A frame {g;};-, satisfying (13) is called a dual frame of { fi }?2,. The hope 
is to find dual frames that are easier to calculate or have better properties than the 
canonical dual. Examples of this type can be found in [17]. 


3 Operators 

In this section, we introduce several operators that play key roles in Gabor analysis. 
In particular, we will need the basic properties of the localized Fourier transform, 
which is called the STFT (short-time Fourier transform). It is natural for us to start 


with the Fourier transform, which is defined as an integral transform on the space 
of all (Lebesgue) integrable functions, denoted by L!(R¢). 


The Fourier Transform 
Definition 5. For f ¢ L!(R@), the Fourier transform is defined as 


Fw) = (F fle) = f flsyePe de, (14) 


where x-@ = Sy X,;@, is the usual scalar product of vectors in R?¢. 
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Lemma 3 (Riemann-Lebesgue). /f f € L'(IR7), then f is uniformly continuous 
and lim|,|->00 | f(@)| = 0. 


The Fourier transform yields a continuous bijection from the Schwartz space 
S(R“) to S(R“). This follows from the fact that it turns analytic operations 
(differentiation) into multiplication with polynomials and vice versa: 


F(D* f) = (Ani) X°(F f) (15) 
and 
D°(F f) = (-2ni)"F(X" f), (16) 
with a multi-index w@ = (q,...,aa) € N¢, la] := y a;, D® as differential 
operator 
D® f(x) = Fer apgar Pea) 
and X® as multiplication operator (X° f)(x) := xf'++-x97 f(x1,....Xa). It 


follows from the definition that S(IR7) is invariant under these operations, i.e., 
X"f €S(R’) and D*f € S(R4) Vae Nt Vf € S(R*). 


Using the reflection operator (Zf)(x) := f(—x), one can show that F? = T and 
oF = Id gga). This yields 


F\=IF (17) 
and we can give an inversion formula explicitly: 


Theorem 5 (Inversion Formula). The Fourier transform is a bijection from S(R¢) 
to S(IR“), and the inverse operator is given by 


Ff) = [ foe"? dw Wx eR?. (18) 
Rd 


Furthermore, 


(FRFe)2 =(fiehe Vhg €S(R’). 


We can extend the Fourier transform to an isometric operator on all of L?(R¢). 
We will use the same symbol F although the Fourier transform on L?(R“) is not 
defined by a Lebesgue integral (14) anymore if f € L?\L!(R@), but rather by 
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means of summability methods. Moreover, F f should be viewed as an equivalence 
class of functions, rather than a pointwise given function. 


Theorem 6 (Plancherel). Jf f € L! L?(R%), then 


If lle = FF lh. (19) 


As a consequence, F extends in a unique way to a unitary operator on L?(R“) that 
satisfies Parseval’s formula 


(fg =(FFFeie Vig eR. (20) 
In signal analysis, the isometry of the Fourier transform has the interpretation 
that it preserves the energy of a signal. For more details on the role of the Schwartz 
class for the Fourier transform, see [78, V]. 
Translation and Modulation 
Definition 6. For x,@ € R®, we define the translation operator T,. by 
(Tx f)(0) = f(t — x) (21) 
and the modulation operator M, by 
(Mo f(t) = e'" F(0), (22) 
One has TT ! — T_. and M oe ! — M_.,. The operator 7. is called a time shift 
and M,, a frequency shift. Operators of the form T, M,, or M,,T, are called time— 
frequency shifts (TF-shifts). They satisfy the commutation relations 
TyMy = €°"*"° Mo Tx. (23) 
Time-—frequency shifts are isometries on L” for all | < p < 00, ie., 
[Tx Mo f lle = If ll. 
The interplay of TF-shifts with the Fourier transform is as follows: 
T.f =M_,f or FT, =M_xF (24) 
and 


Ma f =Tof or FMy = Tif. (25) 
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Equation (25) explains why modulations are also called frequency shifts: modula- 
tions become translations on the Fourier transform side. Altogether, we have 


TM, f = Mit. F = eae ae 


Convolution, Involution, and Reflection 


Definition 7. The convolution of two functions f, g € L'(R®) is the function f * g 
defined by 


(f * g)(x) = / f(y) g@& — y) dy. (26) 
Rd 
It satisfies 


If * sll <i fllligiy: and fxg = f-g. 


One may view f * g as f being “smeared” by g and vice versa. One can thus 
smoothen a function by convolving it with a narrow bump function. 


Definition 8. The involution of a function is defined by 


f* (x) = f(x). (27) 


It follows that 


RK 


£SFf and Tf=Tf. 


Finally, let us mention that convolution corresponds to pointwise multiplication (and 
conversely), i.e., the so-called convolution theorem is valid: 


eel ai-s. (28) 


The Short-Time Fourier Transform 


The Fourier transform as described in section “The Fourier Transform” provides 
only global frequency information of a signal f. This is useful for signals that do 
not vary during the time, e.g., for analyzing the spectrum of a violin tone. However, 
dynamic signals such as a melody have to be split into short time intervals over 
which it can be well approximated by a linear combination of few pure frequencies. 
Since sharp cutoffs would introduce discontinuities in the localized signal and 
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therefore leaking of the frequency spectrum, a smooth window function g is usually 
used in the definition of the short-time Fourier transform. 

In image processing, one has plane waves instead of pure frequencies; thus, the 
global Fourier transform is only well suited to stripe-like patterns. Again, a localized 
version of the Fourier transform allows to determine dominant plane waves locally, 
and one can reconstruct an image from such a redundant transform. Gabor analysis 
deals with the question of how one can reconstruct an image from only somewhat 
overlapping local pieces, which are stored only in the form of a sampled (local) 2D 
Fourier transform (Fig. 1). 


Definition 9. Fix a window function g € L?(R“)\{0}. The short-time Fourier 


transform (STFT), also called (continuous) Gabor transform of a function f € 
L?(R“) with respect to g, is defined as 


(Ve f)(x,@) := / fit)gt—xje2""? dt forx,w € R?. (29) 
Rd 


For f.g € L?(R%), the STFT Y.f is uniformly continuous (by Riemann— 
Lebesgue) on R~@ and can be written as 


(Ve f)(x,0) = f- TB) (30) 
= (f, MoTr8)12 (31) 
= Yeas © J i M.g*)(x). (32) 


The STFT as a function in x and w seems to provide the possibility to obtain 
information about the occurrence of arbitrary frequencies w at arbitrary locations 
x as desired. However, the uncertainty principle (cf. [51]) implies that there is a 
limitation concerning the joint resolution. In fact, the STFT has limitations in its 
time—frequency resolution capability: Low frequencies can hardly be located with 
narrow windows, and similarly, short pulses remain invisible for wide windows. The 
choice of the analyzing window is therefore crucial. 

Just like the Fourier transform, the STFT is a kind of time—frequency represen- 
tation of a signal. This again raises the question of how to reconstruct the signal 
from its time—frequency representation. To approach this, we need the orthogonality 
relations of the STFT, which corresponds to Parseval’s formula (20) for the Fourier 
transform: 


Theorem 7 (Orthogonality relations for STFT). Let fi, ff. 21.22 € L?(R®). 
Then V2, fj € L?(R*4) for j € {1,2}, and 


(Ve. fi, Veo 2) 12 (R24) = (fi, fo)r2(81, 82)12- 


Corollary 3. /f f, g € L?(R®), then 
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Fig. 1 Two signals and their (short-time) Fourier transforms. (a) Signal 1: Concurrent frequencies. 
(b) Signal 2: Consecutive frequencies. (c) Fourier power spectrum 1. (d) Fourier power spectrum 
2. (e) STFT | with wide window. (f) STFT 2 with wide window. (g) STFT 1 with narrow window. 
(h) STFT 2 with narrow window 


IMef lhoges) = Wf hele: 


In the case of |\g||_2 = 1, we have 
If lhe = Melle VF EVR, (33) 


i.e., the STFT as an isometry from L?(R®) into L?(R4). 
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Formula (33) shows that the STFT preserves the energy of a signal; it corresponds 
to (19) which shows the same property for the Fourier transform. Therefore, f 
is completely determined by VY, f, and the inversion is given by a vector-valued 
integral (for good functions valid in the pointwise sense): 


Corollary 4 (Inversion formula for the STFT). Let g,y € L?(R“) and (g,y) # 
0. Then 


fx) = —— ffs v.fee.0) MoT) dodx Vf eLR). G4) 
(Y, 8)L2 JSR 


Obviously, y = g is a natural choice here. The time—frequency analysis of 
signals is usually done by three subsequent steps: 


(i) Analysis: Using the STFT, the signal is transformed into a joint time—frequency 
representation. 
(11) Processing: The obtained signal representation is then manipulated in a certain 
way, e.g., by restriction to a part of the signal yielding the relevant information. 
(iii) Synthesis: The inverse STFT is applied to the processed representation, thus 
creating a new signal. 


A function is completely represented by its STFT but in a highly redundant way. 
To minimize the influence of the uncertainty principle, the analyzing window g 
should be chosen such that g and its Fourier transform ¢ both decay rapidly, e.g., 
as Schwartz functions. A computational implementation can only be obtained by a 
discretization of both the functions and the STFT. Therefore, only sampled versions 
of the STFT are possible, and only certain locations and frequencies are used for 
analyzing a given signal. The challenge is to find the appropriate lattice constants in 
time and frequency and to obtain good time-frequency resolution. 


4 Gabor Frames in L?(Rd) 


By formula (31), the STFT analyzes a function f €L?(R®) into coefficients 
(f, MoTyg),2 using modulations and translations of a single window function 
g € L?(R“)\{0}. One problem we noticed was that these TF-shifts are infinitesimal 
and overlap largely, making the STFT a highly redundant time—frequency represen- 
tation. An idea to overcome this is to restrict to discrete choices of time positions 
x and frequencies w such that this redundancy is decreased while leaving enough 
information in the coefficients about the time-frequency behavior of f. This is the 
very essence of Gabor analysis: It is sought to expand functions in L?(R®) into an 
absolutely convergent series of modulations and translations of a window function 
g. Therefore, it is interesting to find necessary and sufficient conditions on g and a 
discrete set A C R¢ x R@ such that 
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{Sx,w}(xw)ea = {M, Ty Bh (x0)eA 


forms a frame for L?(R“). The question arises how the sampling set A should 
be structured. It turns out to be very convenient to have this set closed under the 
addition operation, urging A to be a subgroup of the time-frequency plane, i.e., 
A <I R¢ x R¢. Dennis Gabor (actually Dénes Gabor) suggested in his Theory of 
Communication [45], 1946, to use fixed step sizes a, B > 0 for time and frequency 
and use the set {wk},¢7a for the time positions and {6n},,¢7a for the frequencies, 
yielding the functions 
Skn(X) = Men Tak 3 (x) = geno — ak) 

as analyzing elements. This is the approach that is usually presented in the literature, 
although there is also a more general group theoretical setting possible where A is an 
arbitrary (discrete) subgroup. This subgroup is also called a time—frequency lattice, 
although it does not have to be of such a “rectangular” shape in general. 


Definition 10. A lattice A C R® is a (discrete) subgroup of R¢ of the form A = 
274 , where 2 is an invertible d x d-matrix over R. Lattices in R24 can be described 
as 


A = {(x, y) € R™ | (x, y) = (Ak + Bl,Ck + D2), (k, 0) € 2} 


with A, B,C, D € C¢*4 and 
AB 
A= . 
& >) 


A lattice A = aZ¢ x BZ4 <I R™ fora, B > 0 is called a separable lattice, a 
product lattice, or a grid. 


In the following, our lattice will be of the separable type for fixed lattice 
parameters a, B > 0. 


Definition 11. For a nonzero window function g € L?(R“) and lattice parameters 
a, B > O, the set of time-frequency shifts 


G(g, a, B) = {Mgr Tuk Shkneza 


is called a Gabor system. If G(g, a, B) is a frame for L7(R®), it is called a Gabor 
frame or Weyl-Heisenberg frame. The associated frame operator is the Gabor frame 
operator and takes the form 


Sf = Dees Men Tak &)12 MonTakg (35) 


knéZé 
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= SO Ve f (ak, Bn) MpnTaks 


k.nez4 
for all f € L?(R). The window g is also called the Gabor atom. 


According to the general frame theory, {S~!gxn}x nega yields the canonical 
dual frame. So we would have to compute S~! and apply it to all modulated 
and translated versions of the Gabor atom g. A direct computation shows that for 
arbitrary fixed indices £,m € Z4, 


SMemT xe = Mp ToS. (36) 


Consequently, also S~'! commutes with time-frequency shifts, which gives the 
following fundamental result for (regular) Gabor analysis: 


Theorem 8. /f the given Gabor system G(g, a, B) is a frame for L?(R¢), then all of 
the following hold: 


(a) There exists a dual window y € L?(R®) such that the dual frame is given by the 
Gabor frame G(y, @, B). 
(b) Every f € L?(R®) has an expansion of the form 


f= > su MenTak&)12 MpnTaky (37) 


k,neZa 


= a bac: MonTaky)12 MpnTak& 


kneZé 


with unconditional convergence in L?(R¢). 

(c) The canonical dual frame is given by the Gabor frame {MgnTak S~' 8} nezé 
built from the canonical dual window y° := S~'g. 

(d) The inverse frame operator S~' is just the frame operator for the Gabor system 
G(y°, a, B) and 


ST f= °F MpnTakv?),2 Mpn Tory? (38) 


kneZa 


We note that if the function g is compactly supported and the modulation param- 
eter f is sufficiently small, it is easy to verify whether G(g, a, 8) is a frame and to 
find the canonical dual window in the affirmative case; see [51, 6.4] or [17, 9.1]. 

One can show [51, 7.6.1] that all dual windows y of a Gabor frame G(g, a, 6) are 
within an affine subspace of L7(IR“), namely, y € y° + K+, where K is the closed 


linear span of G (¢. 3 1) and therefore 


Qa 
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K+ = {he L?(R%): (h, MnjaTk/pg)u2 =0 Ven eZ}. (39) 


Hence, we have y = y° +h fora certain h € K+, and as y° € K, the canonical 
dual window possesses the smallest L?-norm among all dual windows and is most 
similar to the original window g. However, there might be reasons not to choose the 
canonical dual window, but one of the others in y° + K+, if, e. g., one wants the dual 
window to have a smaller essential support or if the window should be as smooth as 
possible. Explicit constructions of alternative dual windows can be found in [17]. 

A key result in Gabor analysis states a necessary condition for a Gabor system to 
form a frame: 


Theorem 9. Let g € L?(IR“)\{0} and a, B > 0. If G(g, a, B) is a frame, then: 


(a) af <1. 
(b) G(g, a, B) is a basis if and only if aB = 1. 


Unfortunately, having wB < | is not sufficient for a Gabor system to form 
a frame. Sufficient conditions are presented, e.g., in [16, 8.4]. A special result is 
known for the Gaussian function: 


Theorem 10. Consider the normalized Gaussian g(x) := 24/4e-**’, Then 
G(g, a, B) is a frame for L?(R®) if and only if aB < 1. 


In signal analysis, it is customary to call the case 


aB <1 oversampling 
aB = | critical sampling 
aB > 1 undersampling 


In the case of the Gaussian window, oversampling guarantees an excellent time— 
frequency localization. But for Gabor frame theory in L?(R®), it is quite delicate 
to find appropriate windows for given wB < 1. The case a6 = 1 is problematic 
from the point of view of time-frequency analysis, as the Balian—Low theorem 
demonstrates: 


Theorem 11 (Balian—Low). Let g € L7(R“) be a nonzero window and a, B > 0 
with aB = 1. If g has good TF-concentration in the sense of 


|Xgll2IXll2 < o9, 


then G(g, a, B) cannot constitute a frame. 
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Combining Theorems 9 and 10 shows that it is impossible for a Gabor basis to 
be well localized in both the time domain and the frequency domain. This motivates 
the study of redundant Gabor systems: As demonstrated by Theorem 10, redundant 
Gabor frames exist for any a6 < 1. 


5 Discrete Gabor Systems 


For practical implementations of Gabor analysis, it is essential to develop discrete 
versions of the theory for Gabor frames. 


Gabor Frames in £7(Z) 


Classically, most signals were considered as “continuous waves.” Indeed, the 
technology for signal processing originally was of the continuous-time analog type 
before digital computers came into our everyday life. Nowadays, digital signal 
processing is used almost exclusively, forcing us to change our function model to a 
time-discrete one. It is therefore natural to switch from L?(R) to £?(Z). 

Gabor frame theory in ¢7(Z) is very similar to that in L?(R) and will therefore 
only be discussed briefly in this section. The main differences concern the time 
shifts and frequency shifts. Time shifts are given as multiples of integer translates, 
ie., 


Le) =F =) (40) 


fork € Zand f € £?(Z). A shift parameter w > 0 for Gabor frames in £7(Z) can 
only be givenasa = N EN. 

For fixed L € N and corresponding to the modulation parameter 1/L, we define 
the modulation operator My by 


MifG)ser 7) (41) 


for £ € Z. Modulations are now periodic with period L, ic., Me+ni = 
M; ‘Vn &€ Z, implying that one needs only the modulations Mo,..., Mz-1. 

The discrete Gabor system generated by the sequence g € €7(Z), shift 
parameters N, and modulation parameter 1/L is now the family of sequences 


{8ketkez,ce(L) Where 
gne(J) = MeTkvg() = ee ag —kN) 


and (L) := {0,...,L—1} CZ. 

If a Gabor system satisfies the frame inequalities for f ¢ ¢7(Z), the dual frame 
is again a Gabor frame built from a dual window y € ¢7(Z). The frame expansion 
takes the form 
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co «€6L-1 
ae ® (f, MeTeny)2 MiTewg for f € €(Z). 
=0 


ir 
| 
8 


Many results and conditions for Gabor systems in £7(Z) can mutatis mutandis be 
taken over from L7(R), e.g., a necessary condition for the mentioned Gabor system 
to be a frame for £7(Z) is that aB = N/L <1. 

We note that there is a natural way of constructing Gabor frames in ¢7(Z) from 
Gabor frames in L?(R) through sampling; see the paper [55] by Janssen. 

The step from L?(R) to €?(Z) is the first one toward computational realization 
of Gabor analysis. However, since in finite time only finitely many elements can be 
considered, only vectors of finite length and finite sums can be computed. Therefore, 
we turn to signals of finite length next. 


Finite Discrete Periodic Signals 


In practice, one has to resort to finite, discrete sequences. We will consider signals 
f € C4, ie., signals of length L € N, and write f = (f(0),..., f(L—1)), 
defined (for convenience) over the domain (L) := {0,..., L— 1} C Z. This way of 
indexing suggests in a natural way to view them as functions over the group of unit 
roots of order L or equivalently as periodic sequences with 


fG tnLl):= fGi) Wane Z,j €(L). 


The discrete modulation M; defined in (41) can still be applied; the translation 7; 
defined in (40) can be taken from the range 0 < k < L—1. 
The discrete Fourier transform (DFT) of f € C% is defined as 


L-1 


AJ := FAG = VfB, Fey, (42) 


k=0 
which is — up to a constant — a unitary mapping on C°. Its inverse is 


L-1 


FC/G) = pLiwe BIE 5 Eline (43) 


The unitary version C’ —> C% has the factor 1/ \/L in front. A well-known and 
very efficient implementation of the DFT is the fast Fourier transform (FFT). 

The discrete STFT of f € C” with respect to the discrete window g € C% is 
given as 


Ve fk. 8) = (f MT 8) cr 
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The actions of time and frequency shifts are in more detail given as 


Tif =Ti(f),.... f(L-)) = (CW... fE-1-) 


and 


Mf = Mi(f(),.... f(L—) 
= (f(0), 2/4 F(1), eb" 4/F £(2), ae CPE ME OL — 1). 
The actions of the TF-shifts can be described as matrices that operate on the vector 
f = (f(O),..., f(L —1))'. The time-shift matrix 7; is given as the permutation 
matrix with ones on the (periodized) k-th subdiagonal, whereas the modulation 


matrix has its exponential entries positioned at the main diagonal. It is obvious that 
the composition of arbitrary TF-shifts need not be commutative, since 


TMe = PE MT, =k, € € Zp 
To get a more compact notation for TF-shifts, we write 
(A) := wk, €) = MeT, with A= (k, 0) €Z, x Zz, 


where Z, x Z, is the discrete time-frequency plane. The commutation relations 
imply for A = (r,m) and pp = (s,n) 


m(A) (pw) = (A+ py err’ (44) 
m(|L) m(A) e2ti(rn—sm)/L (45) 


II 


Frames and Gabor Frames in C/ 
The general frame definitions and results can easily be carried over to the case 
of finite discrete signals. The conditions for the finite sequence {go,..., gv—1} of 


elements g; € C* to be a frame for the finite-dimensional Hilbert space C“ are that 
there exist A, B > 0 such that 


L-1 N-1 L-1 
AY“ |f@? <> |(heneel < BY IP vf ect 
k=0 j=0 k=0 


or 


All fll < ICSF Ib < BIS IIa VF eC, 


Gabor Analysis for Imaging 1737 


where C is the analysis operator. It is obvious that the sequence {g; yo has to 
span all of C*, ie., spantgj PSs = C’;hence, N > L ina Hilbert space with 
dimension L. Also the converse is true: Every spanning set in C“ is a frame for C’. 

The action of a linear analysis operator C on the vector f is given as the vector 


Cf = ((f, gj ye yeas ' indicating that its j-th entry is 


L-1 


(Cf); = (f.a;) = of e®. 


k=0 


Letting g* = g, the matrix form of C € C%** is 


> 800) te aunt) 


SN-1 &n- Oe - ZN- TL —1) 


A family {g;}je(w) is a frame for C* if and only if the corresponding analysis 
operator C has full rank, and every matrix with full rank uniquely represents a 
frame. 

The frame operator S = C*C becomes an L x L matrix that also has full rank, 
and it is therefore invertible. Its condition number equals the ratio between its largest 
and smallest eigenvalue; letting A denote the largest lower frame bound and B the 
smallest upper frame bound, this is equal to the ratio B/A. 

If we translate the discrete frame expansion 


c(0) PBT O03) 
f =C*c = (g0,...,8Nn-1) : = ‘ 
c(N-DJ AD egi-) 
fora given f € C¥, we see from a linear algebra point of view that we are looking 


for N unknown coordinates of c € C%, using L < N equations. Clearly, the 
solution cannot be unique if L < N. Considering that 


f=SS'f =c*c(c*cy'f 
we see that one solution for c could be given as 
e=C(C*Cy'f =(C*)f 


in terms of the pseudo-inverse of the synthesis operator C’*. This also provides the 
matrix form of the canonical dual frame that is given by 


(S~'g0,...,S7!gw-1)" = (S7!c*)* =cs7! =(C*)}. 
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We will now proceed to the special case of Gabor frames. They are given as a 
sequence of TF-shifts of a single window function g € C%, i.e., a Gabor frame for 
C# is a sequence {gy},ceq := {m(A)g}aca for a certain discrete subset A C Z;, x 
Z. We write C, for the Gabor analysis operator to indicate the dependence on g and 
use it synonymously for the Gabor frame itself. It is clear that it is necessary to have 
N > L elements to span all of C*, but this is of course not sufficient for validating 
a frame. The ratio between N and L is also called the redundancy of the frame, 


N 
dc i= —. 
redc 7 


For any subgroup A <I Z; x Z,, the Gabor frame operator Sy = CiC, 
commutes with all TF-shifts 7(A) for A € A. This can be shown in a similar way as 
in Sect. 4. Therefore, the dual frame is once again a Gabor frame, built by the same 
TF-shifts of a single dual window y € C*. The canonical dual frame consists of 
elements 


Sy'mA)g = aQ)Sy'g = 2(A)y®, 


and the computation of the canonical dual window reduces to finding a solution for 
the linear equation 


Sey? = g. (46) 


Therefore, the discrete Gabor expansion of an f € C* is given as 


f= (Aaa) TAY? = (FAY )or TADS. 


AEA AEA 


where the Gabor coefficients belong to £7(A) = C%. 
A special case for a lattice is a so-called separable lattice A = aZ, x BZ, with 
a, B € N being divisors of L. The elements of such a Gabor frame take the form 


MpeToxg(j) = e184" g(j — ak) 


with k € (4) andf¢ (4). The number of elements is N = re = Le, and 
it is necessary to have 5 > L elements to have a frame. The oversampled case 


is therefore given for a6 < L, and the undersampled case for a6 > L. Critical 
sampling is given for #6 = L. 
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6 Image Representation by Gabor Expansion 


We have seen that Gabor analysis can be considered as a localized Fourier analysis, 
where the main design freedom is the choice of (a) the time-frequency lattice and 
(b) the window function. The type of sampling lattice can be distinguished into a 
separable or non-separable case, where the first one can be described by the choice 
of lattice constants a, B > 0. 

It turns out that in the twofold-separable case, i.e., where the d-dimensional 
analysis window is a tensor product of d one-dimensional functions 


= 8, 8--Oga, with gy @---@ ga(x,...,Xa) = gi(%X1)... gala), 


and the sampling lattice A is a product A = Mt, a; Zp, X Tes B;Zz,, the dual 
Gabor window y is given as a product y = y; ®--- ® yg as well. Thus, the 
computation is reduced to finding the 1D duals y; of the 1D atoms g; with respect 
to the corresponding 2D time-frequency lattices Aj = a; Z,, x B;Z1,. 

Our aim here is to show how the results can be applied to the case of image 
signals. Gabor expansions of finite discrete 2D signals (i.e., digital images) are 
similar to those of finite discrete 1D signals, and in a more general notation, there is 
no difference at all. We are going to describe it next (Fig. 2). 


2D Gabor Expansions 


The key point for the development of efficient algorithms is to interpret an image of 
size L, x Lo as a real- or complex-valued function on the additive Abelian group 
G =Z,, x Z_,. The position—frequency space is 


—~ Pm marca 
GxG=Z,,xZ1,xZz, x Zr). 


0.06 


—0.04 


Fig. 2. Typical 2D Gabor 
atoms 


—0.06 
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A Gabor system G(g, A) consists of TF-shifts M;T,g of a window g € ChixL2 | 
where (k,l) are elements of a sampling subgroup A <l Z,, x Zi, x Zz, x Z,. The 
Gabor coefficients of the image f ¢ C/'*"? are defined as 


cer = (f, MiTig)p, (kD € A. (47) 


Here, we use the subscript F in order to recall that for matrices (this is how images 

are usually stored), one takes the scalar product and the corresponding norm just as 

the Euclidian one in C’, with N = LiL, usually denoted as Frobenius norm. 
The Gabor system is a frame if for0 < A < B < oo one has 


Alp < SU MiTigyel? < BIZ vf e CE. 
(kDEA 


For dimensionality reasons, it is clear that the frame condition is only possible if 
the number of elements in A has to be at least equal to the dimension of the signal 
space, and, therefore, we need L;L2 < |A| < (L1L2)?. The redundancy of the 
Gabor frame is 


|A| 
LiL, 


red, := > 1. 


As in the one-dimensional case, the Gabor frame operator 


f= » (f, MiTg)e MiTkg 
(k.DEA 


commutes with TF-shifts determined by A, and the minimal resp. maximal eigen- 
value are equal to the maximal lower frame bound A and minimal upper frame 
bound B, respectively. 

Again, the dual Gabor frame has a similar structure as the Gabor frame itself: 
using the same TF-shifts, now applied to a dual window y € C/!*!2, one has the 
expansion 


f= Yo (f.MiTig)e MiTey = S > (Of. MiTey)e MiTeg 
(kDEA (kDEA 


for all f ¢ C!'!*!2, The existence of the dual atom is guaranteed by the theory 
of frames, and the calculation of the dual Gabor frame is done by the methods 
developed there. Recent results guarantee that good TF-concentration of the atom g 
implies a similar quality for the dual Gabor atom. Typically, the condition number 
of the frame operator depends on the geometric density (hence, to some extent, 
on the redundancy) of the lattice. However it is worth mentioning that even for 
low redundancy factors, relatively good condition numbers can be expected for 
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g=gl.’*g2 gd = gd1.’ * gd2 


Fig. 3 2D separable window and its dual on a fully separable lattice. (a) Lattice Ay = 15 Zj29 X 
6Z20. (b) Lattice Ay = 8 Zj6o X 16Z 6. (c) Guassian g1 and its dual Vr on A,. (d) Guassian g2 
and its dual yy on A». (e) 2D window g=g1 ® g2. (f) 2D dual window y° = yp @ y; on Ay X Ag 
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suitably chosen atoms and that perfect reconstruction can be achieved in a stable 
way in a computationally efficient way even if the discretization of the continuous 
representation formula is far from satisfactory. Expressed differently, the frame 
operator may be far away from the identity operator but still stably invertible. 

The optimal method and effective computational cost for obtaining Gabor 
expansions of an image depend on the structure of the 4D sampling lattice. A (fully) 
separable position—frequency lattice (PF-lattice) can be described by parameters 
1, @2, B}, By > O such that the constants a; and 6; describing the position and fre- 
quency shift parameters are divisors of L;, respectively. The set A itself is given as 


A= Kua) = (ky, ko, 1, €2) = (aim, 22, Biv1, Brv2) | ui € (2), € Cale 


Le., it is a product group: A = A, x Az with A; = a; Zz, x BiZ1,- 

Full separability may be violated in different ways. Assume that A = A; x Aj 
but with non-separable 2D lattices A;. There are at least two natural choices, whose 
usefulness may depend on the concrete application. The first and probably more 
relevant choice is a lattice A, in position space and A», another lattice, in the wave- 
number domain. For the case of radial symmetric windows, g, one may choose a 
hexagonal packing in both the spatial and the wave-number domain. 

Another flavor of separability comes in by choosing lattices within C“i and ch, 
respectively, describing the first and the second pair of phase space variables. 

In passing, we note that there are also fully non-separable subgroups. They 
will not be discussed here, because it is not clear whether the increased level of 
technicality is worth the effort. 


Separable Atoms on Fully Separable Lattices 


In this section, we will show why the case of a 2D separable window g = gj ® g2 
and a fully separable PF-lattice 


A= A,x Ap = Zz, x BiZL, x eZ, x BoD, 


allows for very efficient Gabor expansions at decent redundancy. It is crucial to 
observe that in this case, it is enough to find a dual 1D window y; for the 1D window 
g, on the TF-lattice A; <I Z,, x Zi, and a dual 1D window y, for the 1D window 
g2 on the TF-lattice Az < Zy, x Zi, in order to obtain a dual 2D window y for 
g for the lattice A, simply as y := y; ® y2. In short, the 2D Gabor frame on the 
product space C/! @ C/? is obtained by combining via tensorization the Gabor 
frames for the signal spaces C“! and C’?. The abstract result in the background can 
be summarized as follows: 
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Lemma 4. [f{@m}me(m) ch and { fntne(no) S C2 are frames for C™ andC!, 
respectively, then the sequence {@m ® fnt(mn)ye(Ni)x(No) 18 a frame for Ch @ Cc, 
where (g @ h)(j,k) := g(j) h(k) for g € C!! andh € C". The joint redundancy 


; 12 > 
is TT > 1. 


As our image space is a tensor product, we define 2D Gabor windows g € C/!*/2 
by g = g1@& for g; € C“’. As we are looking at the case where A = A; x Ap, we 


take two Gabor frames {2(?,,} ay = {Me Th; Bi} (ej t:yeni C Ci with frame 
PNT (kj i )EAi 
(2) 


operators S; and use the set of products {ste ® 8.6 Cc C41 @C as 


k DEA 
frame for the image space with frame operator S; @ Sp. = 

In order to ensure the fact that this is a 2D Gabor family, one just has to verify that 
the translation by some element in a product group, applied to a tensor product, can 
be split into the action of each component to the corresponding factor. Finally, the 
exponential law implies that a similar splitting is valid for the modulation operators; 
in fact, plane waves are themselves tensor products of pure frequencies. We thus 
have altogether 


Me, Tk, 81 ® Me Tk 82 = Mee, 0) Ttk.ko) (81 @ 82) Wki, ko), (C1, 2) € Zr, x Zz, 


as building blocks for our 2D Gabor frame. 
The canonical dual of g with respect to that frame is given as 


y°=S ‘g= 8; 'g1@ S;'g =, @yp. 


The calculation of 1D dual windows for separable TF-lattices has been efficiently 
implemented in MATLAB available from the NuHAG webpage (http://www.univie. 
ac.at/nuhag-php/mmodule/resp. by Peter Sgndergaard LTFAT (linked with the 
above page)). 

Next, let us check out how we can efficiently obtain the Gabor coefficients of an 
image f € C/'*/? as given by (47). How does the Gabor matrix C, look like if it 
is to be applied to an image f ¢ C/!*/2 stored as an L, x L2-matrix? For sure, f 
must be seen as a vector in C/1“2 and C, as an N; No x L, L2-matrix if the number 
of elements in the 2D frame is N; N> and the coefficient vector is c € C™!4?. In 
general, f cannot be assumed to be separable; thus, the only thing simplifying our 
computation is the structure 


cea = (f, Me, Tk, 81 ® Me, Tk 82) ¢- 


If we think of the 1D case with some f € C” anda general frame {g;}j<(w) CC’, 
the coefficients are obtained by 


c= Cf = ((£.8;)) jew = ise) 
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and for Gabor frames, c = (cx,¢)(k,2)ea With A <I Zy, x Zs is actually a coefficient 
matrix in C/** with |A| = N < L? nonzero entries. But due to simply stacking 
the vectors {gch(k.Qen = {8st je(n) S C# in the coefficient matrix 


8 
c=| : Jecr™, (48) 
SN-1 
one just gets a “flat” c € C. In our 2D case, the Gabor coefficient even consists of 
entries ChI = Ck, ,ko,l;,€.- We also want to take the approach by using general frames 
{8m}me(n) SG C/! and {hn}ne(ny) © C!2 and look at the product frame { gn @/n} inn 


for C4! @ C2. We also reduce the coefficient c = (c(m, 1))mnn © C42 to a vector 
of the form 


c= (c(0, 0), c(0, 1),...,c(0, N2 — 1),c(1,0),...,c(1, No —1),... 
...,€(N; —1,0),...,¢(Ni — 1, No —1))" 


such that we can try to find the corresponding coefficient matrix C € CN142*4i/2 
that can be applied to f ¢ C/'?, where 


f =(f@,0),..., f,L2—-1), f0,0),..., f@i-1,L2-D)". (49) 


Now, we can look at the (m, m)-th or, rather, (mN2 + n)-th entry of the coefficient: 


(Cf) mn = c(m,n) (f &m ® hyn) clits 
Ly-1 L2-1 


Y> YS FY) Gn Blin )u v) 


u=0 v=0 


II 


Li-1 L2-1 


Yo dS PUY) gn) Tn). (50) 


u=0 v=0 


II 


Since we are now able to split the indices u and v for the frame elements, we can 
consider the order in (49) and get 


(Cf) mn = (gm (0) h* &m(1) h* a Smn(Li = 1) h*) if = (C)innf 


where (C),.» 1s the (m, n)-th or (N+ n)-th line of C and contains L; L2 entries. 
The line vectors {ha }netn,) form the frame matrix C) € C42**2 like in (48). If we 
look at the range of N2 lines {(m,0),..., (mm, N2 — 1)}, we are able to express the 
corresponding segment of C as 


(C )insne(no) = (gm (0) C2 &m(1) C2 a En(Li = 1) C2) : 
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This shows that the frame matrix of the product frame is the Kronecker product of 
the partial frame operators C; € CMXLi j= 1,2: 


C=C OG 6 CNY, 


Nevertheless, we want to see whether we can compute c = (C; @ C2) f ina 
cheaper way by applying the frame matrices C; without computing their Kronecker 
product. As images are not stored as vectors f €¢ C/'/2 but rather as matrices f € 
C41*/2 in numerical software like MATLAB or Octave, we could try to get the 
coefficient c = (c(m,n)),, , € CN'**2 more directly. 


Proposition 1. Given two frames {8m}me(n,) © C" and {hy ne(No) S C!2 with 
frame matrices C; € CNi**i, then the frame coefficient ce € CN'*%? for the 
image f € C"'*"2 with respect to the product frame {8m ® hn}onnyis given by 
matrix multiplication as follows: 


c=C, xf x cy — 
go(0) - — go(Li-1) ( f(0.0) + ~~ f(0,L2—1) ho(0)  — wy—1(0) 
gy,—10) ~ gnj—ii-) J \ f(Li-1.0)  fi-1.Lo-D J \ figEg=T) tym a=) 


Note that similar thoughts reveal the fact that the 2D DFT of an image f € 
C/1*!2 can be obtained by the matrix multiplication 


Ff = Fr, *f * Fr, € CX, (52) 


where Fy; € Cti*+4i are the (symmetric) Fourier matrices of order L;. 

If the synthesis operation is to be done by f = C*c for given f € C/ 
and a frame C € C%%“, one solution is obtained by c = (C*)'f with a right 
inverse for C* such thatI, = SS~! = C*C(C*C)! = C*(C*)', making the 
pseudo-inverse of the synthesis operator the matching analysis operator. C*(C*)' 
is the orthogonal projection onto the range of the desired synthesis operator. One 
notices that due to (C*)’ = (C‘)*, we already have I, = (C'C)* = C'C, the 
orthogonal projection onto the range of ran C*. Thus, the role of the operators can 
be interchanged, meaning that Ct is the matching synthesis operator for the analysis 
operator C. 

If we again interpret signals f ¢ C4! @ C’? as f € C/'/2 and take a product 
frame {2m @ hn}mn With analysis operator C; ® C2, we get Iz,,, = C1(C; @ Co) 


andI,,7, =Iz, @Ik, = (c/a) ® (cio), yielding that the matching synthesis 


operator is CT = ci ® Cc . Due to Proposition 1, we can thus reconstruct f € 
CLixL2 by 
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T 


f= (cia)F (cla): =Cle (cZ) (53) 


because c = C; fC} is in the range of the corresponding analysis operator. 

These results were derived for products of general frames and therefore also hold 
for products of Gabor frames. Given two Gabor frames {Mp, T;,, g;} on subgroups 
A; IZ, x Zit, and with analysis operators C,,, we get their synthesis operators by 
ch = Cre with y? := So 'g;. The product of those two frames is the Gabor frame 
{MiTkg} cnea,;xA, consisting of PF-shifts of the window g = g; ® g2 € Chixla 
on the lattice A = A; x A. The dual window to g is given by y® := y? @ y3. 
Due to (51) and (53), the 2D Gabor analysis operation for the image f € Chix is 
obtained by 


c=, FC. (54) 


and a possible reconstructing synthesis operation by 


fC (Gay =C,p eC 


i (95) 
yielding that it is enough to obtain the two duals y?. Figure 3 shows the construction 
and look of the separable dual 2D window of a 2D Gaussian window on a fully 
separable PF-lattice. 


Efficient Gabor Expansion by Sampled STFT 


In the case of a separable 2D atom and a fully separable PF-lattice, we can make 
use of any fast 1D STFT implementation (cf. the NuHAG software page or the 
LTFAT by Peter Sgndergaard) to obtain the Gabor analysis coefficiente = Cy, fC i 


T 
and the Gabor reconstruction f = Cyoe (ee) for a given image f € ChixL2, 


These matrix multiplications from the left and right could still be rather expensive, 
so one can obtain the set of Gabor coefficients ¢ by calculating a finite number of 
sampled 1D STFTs, with the sampling points determined by shift parameters a, a2 
and modulation parameters 61, 62. 


If we remember the 1D case, the Gabor frame C, for C’ by a window g € C/ 
involves a separable lattice A = aZ, x BZ, with |A| = N = x, and for arbitrary 


f €C#, we have 


L-1 


(Ce fee = cee = (f MpTaxg)ce = df) MpiTax@W) = Ve f (ak, BO 


u=0 
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fork € (4) and £ € (4). which can be viewed as a vector of length N if the frame 


is seen as a matrix Cy € CNL In the 2D case, if we consider f = (fo,..., fi,-1) 
with fj := (f(0,7),..., f(li - 1, /))", then b; = C,, f; acts as the Gabor 
analysis operation for all f; € C“' with coefficients b; ¢ C™! forall j € (L2). The 
operation b = C,, f collects these in a matrix b = (bo, ..., b,—1). If we express its 
k-th line as a line vector g; := (b)x = (bo(k),...,br,-1(k)), we get 


dy 


Caf=b=g=| : | eck, 


T 
Qn,-1 


The complete 2D Gabor analysis operation is thuse = g'C, = (C,,g)", and this is 
just the Gabor analysis operation of the vectors gx, € C’? for k € (N,) with respect 
to the Gabor frame C,,. 

All in all, the 2D Gabor analysis operation in the twofold-separable case can be 
obtained by first computing L, 1D STFT-operations of output length N, using the 
parameters a), 6; followed by N; 1D STFT-operations of output length N>2 using 
the parameters a2, B. 

As the reconstruction (2D Gabor expansion) is just a multiplication of the dual 
Gabor matrices C*, from the left and right of c, this task can be seen as a sequence 
of 1D Gabor expansions and can thus be obtained by a sequence of inverse 1D 
STFT-operations as well. There are again two ways: The first one is to do Ny 
inverse operations with output length L» using the parameters a2, B2 followed by 
N> operations with output length L; using a), 6). The second way exchanges L; 
and N; correspondingly. 


Visualizing a Sampled STFT of an Image 


So far, we have visualized the full STFT of an image as a large block image, where 
either each block fully represents the frequency domain and the position of the 
blocks the position domain, or vice versa. As such an image would become rather 
huge, we prefer to visualize only a sampled STFT instead. In the case of a separable 
atom, this can be realized by obtaining the discrete 2D Gabor transform by (54), 
where the two involved matrices Cz, consider a special order of their Gabor frame 
elements Mp, Tx, 9. 

For a Gabor frame {MeTxg}(c.Qea © C* given by a 1D window g € C* on 


a separable lattice A = aZ, x BZ, with N = |A| = is elements, we say that 
the Gabor frame elements are ordered by modulation priority if the frame matrix 
C, € C’** is of the form 
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Fig. 4 Discrete 2D Gabor transform of a zebra, modulation priority. The picture shows the 
absolute values of c = C, fC, 1 where g is the 1D Gaussian of length 480 and Cy is the Gabor 
matrix for the lattice A = 10 Z4g9 X 6 Z4go, whose entries were ordered with modulation priority 


MoTog* 
MpTog* 


Mz /p—-1Tog* 
C,; = MoT\g* 


M1/-1T18* 
M1z/p-1TL/o-18* 


We call it ordered by translation priority if it is of the form 
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Fig. 5 Discrete 2D Gabor transform of a zebra, translation priority. The picture shows the absolute 


values of ¢ = C fC e , where g is the 1D Gaussian of length 480 and C g¢ is the Gabor matrix for 
the lattice A = Z4go X 60 Z4g0, whose entries were ordered with translation priority. The Gaussian 
blurred image in the middle has been scaled into the colormap individually 


MoTog* 
MoTug* 


MoT L/o-18* 
Cc; MiTog* 


Mi Ti jo-18* 


M1 /p-1TL/o-18* 


Obviously, C, = PC, for a suitable permutation matrix P € C‘*%, 
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Fig. 6 A non-separable window and some duals on fully separable lattices. The lattices A; are 
that of Fig. 3. The lattices A/ exchange a; with B;. The last lattice has vertical redundancy 1 and 
horizontal redundancy 6.4. (a) Non-separable window g. (b) Dual y° on A; X Az. (ec) Dual on 
A‘, x A4 with exchange parameters (d) Dual at vertically critical redundancy 


If we take an image f € C/'!*/2 and two Gabor frames {Mo, Tx, gi}, (ki, €i) € Ai, 
on separable lattices A; = a;Z,, x B;Z ,, we can take their product Gabor frame 
for C/'*42 and obtain the mentioned two possibilities for an STFT block image by 
either considering the frame matrices Cy, or C 'e;- Lhe matrices C ‘Bi are ordered by 


modulation priority, and if¢ = Cy, fC a then c consists of a x F, blocks 


Xin kg t= ((f, Me, 02) T(k.k2)8)) ¢, 6, 
such that 


Xoo t+) X0,b5/00—-1 


X14 /o—-1,0 ound X17 /a,—1,L2/o—1 


The blocks X;, x. equal the part (Vef (ki, k2, 1, £2))¢, ,, of the sampled STFT and 
thus contain the whole (sampled) set of frequency shifts for a certain position shift 
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of the window g = g) ® go. The (sampled) frequency domain is therefore spanned 
in each of the blocks X;,,, and their positions in ¢ span the (sampled) position 
domain. Each Xx, ;~, could be seen as a sampled “Fourier image” of the discrete 
Fourier transform f- Tk) ko): 

In the other case, where we have ¢ = Cc. f Cc _ , the Gabor coefficient consists of 
cL x £2 blocks 


Yoo = (V, M (ey 02) T(k.42)8)) gy ks 
such that 


Yoo «t+ Yous/p.—-1 


cw 
ll 


Y14/Bi-1,0 *** YL1/Bi—1,L2/p2—-1 


Here, the blocks Y¢, .¢, equal the part (Vef (ky, k2, £1, £0) ) i ks of the sampled STFT 
and contain the corresponding set of position shifts for a certain frequency shift of 
g. The position domain is spanned in each of the blocks Y;, ¢,, and their positions in 
¢ span the frequency domain. 

Figures 4 and 5 show examples for both cases using the zebra test image. As it is 
a square image, we can take g; = go and thus C,, = C,,. The first figure composes 
the Gabor transform coefficient matrix as blocks of Fourier images. Clearly, the 
overall image reflects the shape of the zebra. The “pixels” of that image contain 
“Fourier jets” that are orthogonal to the edges at the corresponding position in the 
original zebra image. Thus, the “jets” are oriented horizontally where, e.g., the body 
of the animal shows vertical line patterns. The second figure shows blocks of zebra 
images that have been convolved with modulated Gaussians. The absolute values 
show the peaks as black spots within the respective image blocks. 


Non-separable Atoms on Fully Separable Lattices 


Non-separable windows are those that can only be defined considering the com- 
plete image domain Z,, x Z,,, and not Z,, and Z,, separately. These cannot 
be described as a tensor product g; ® g2 with g; € C/' anymore, but only 
generally as g € C!'*/2, With this case, we lose the ability to consider two 
(1D) frames independently for each dimension, and we cannot apply two frame 
matrices independently to an image. It appears that we have to stick to the known 
factorizations of Gabor matrices on (fully) separable lattices with parameters a;, B;, 
and we thus cannot make use of the equidistantly sampled 1D STFT. However, 
under certain conditions, this case can be completely referred to a 1D case, as we 
will see below. 

Figure 6 indicates an important thing about the redundancy. Sure, a redundancy 
of 4H > 1 is only a necessary condition, but it seems to be important to consider 
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the redundancy in each dimension. The involved window is a 2D Gaussian window 
g € C!x160 stretched vertically by $, shrunken horizontally by 3, then rotated 
(counter-clockwise) by am. Figure 6d shows its dual on a fully separable 4D PF- 
lattice with overall redundancy 6.4. It was computed in the work of P. Prinz which 
makes use of the Gabor matrix factorizations [66]. But although the redundancy 
value gives the impression to be safe, it hides the fact that the involved lattice 
is actually A = 10Zj.9 x 12 Zy29 x 5 Zi60 X 5 Zio, yielding the redundancy as 
We . oe = 1-6.4. This shows that the vertical redundancy is critical, and the dual 
has a bad localization in the vertical dimension. It is therefore necessary to make 
sure that the redundancy is reasonably distributed among the dimensions. In this 
sense, fully separable 4D lattices can always be considered as a product of two 2D 
TF-lattices with independent redundancies, no matter what structure the 2D window 


possesses. 


7 Historical Notes and Hint to the Literature 


Nonorthogonal expansions as proposed by D. Gabor in his seminal work [45] of 
1946 were ignored for a long time by the mathematical community. The question 
to which extent the claims made by D. Gabor could be realized in the context of 
generalized functions, was carefully analyzed by A.J.E.M. Janssen in 1981 [53]. 
Around the same time, M. Bastiaans explored the connections between Gabor 
theory and optics [3-7]. In the critically sampled case, he suggested to use the 
biorthogonal function y in order to calculate Gabor coefficients. The connection to 
the biorthogonality relations for dual Gabor windows was pointed out in two papers 
in 1995 [26,54] and brought to the multidimensional case in [39, 40, 69]. 

Two early papers in the field, authored by J. Daugman and Y.Y. Zeevi and 
his coauthors, established a connection between a 2D version of Gabor analysis 
and early vision [27, 46, 64, 83, 84], Various subsequent papers emphasized that 
a Gabor family is not an orthogonal system and that, therefore, computation of 
coefficients has to be computationally expensive. We know by now that while linear 
independence is indeed lost, the rich covariance structure of the Gabor problems 
actually leads to efficient algorithms. 

The mathematical theory of Gabor expansions was promoted in various direc- 
tions in the last two decades. Although a lot of Gabor analysis is naturally valid 
in the context of general locally compact Abelian groups, a substantial body of 
references only covers the standard case, for 1D signals and separable lattices. 

Of course, the theory underlying image processing is formally covered by 
the theory of Gabor analysis over finite Abelian groups as described in [40]. 
Some basic facts in the general LCA context are given in [50], and some further 
results generalize to this setting, applying standard facts from abstract harmonic 
analysis [44]. 

Multidimensional, non-separable lattices are discussed in [41], and [38] deals 
with situations where the isomorphism of 2D groups with certain 1D groups helps 
to use 1D Gabor code to calculate 2D dual Gabor windows. 
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Numerical methods for Gabor representations have been discussed since the first 
and pioneering papers (see, e.g., [2,46, 84]). There are also hints on how to perform 
parallel versions of the Gabor transform [30]. A partial comparison of algorithms is 
in [67] and in the toolbox of P. Sgndergaard. It can be expected to provide further 
implementations and more details concerning numerical issues in the near future. 

One of the most natural applications (based on the interpretation of Gabor 
coefficients) are space-variant filters. Given the Gabor transform, one can multiply 
them with a 0/1 function over the coefficient domain, passing through, e.g., higher 
frequencies within regions of interest, whereas otherwise, only low frequencies are 
stored, thus representing foveated images (with somewhat blurred parts outside the 
region of interest). 

Since different textures in different regions of an image might also be detected 
using Gabor coefficients, natural applications are texture segmentation (see e.g., 
(31, 77]), image restoration [19,82], and image fusion [68]. The extraction of direc- 
tional features in images has been considered recently in [48]. Other contributions 
to texture analysis are found in [49]. Other applications are pattern recognition [76], 
face identification as described in [70], and face detection [52]. 

Some of the material presented in this paper can be found in an extended form in 
the master thesis of the last named author [63]. 


Cross-References 
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This chapter describes a selection of models that have been used to build 
Riemannian spaces of shapes. It starts with a discussion of the finite-dimensional 
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space of point sets (or landmarks) and then provides an introduction to the more 
challenging issue of building spaces of shapes represented as plane curves. A 
special attention is devoted to constructions involving quotient spaces, since 
they are involved in the definition of shape spaces via the action of groups of 
diffeomorphisms and in the process of identifying shapes that can be related 
by a Euclidean transformation. The resulting structure is first described via 
the geometric concept of a Riemannian submersion and then reinterpreted in 
a Hamiltonian and optimal control framework, via momentum maps. These 
developments are followed by the description of algorithms and illustrated by 
numerical experiments. 


1 Introduction 


The analysis of shapes as mathematical objects has constituted a significant area of 
interest in the past few decades motivated by the development of image acquisition 
methods and segmentation algorithms, in which shapes could be extracted as 
isolated objects. Shape analysis is a framework, in which a given shape is considered 
as a single (typically infinite dimensional) variable, requiring the development of 
new techniques for their representation and statistical interpretation. This framework 
has found applications in several fields, including object recognition in computer 
vision and computational anatomy. 

The example in Fig. | can help in framing the kind of problems that are being 
addressed and serve as a motivation. These shapes are fairly easily recognizable for 
the human eye. They do however exhibit large variations, and a description in simple 
terms of how they vary and of how they can be compared is a much harder task. It is 
clear that a naive representation, like a list of points, cannot be used directly, because 
the discretized curves may have different numbers of points, and no correspondence 
is available between them. Coming up with quantitative and reliable descriptors that 
can be, for example, analyzed in a rigorous statistical study is, however, of main 
importance for the many applications, and the goal of this chapter is to provide a 
framework in which such a task can be performed in a reliable well-posed way. 


2 Background 


During the past decades, several essential contributions have been made, using 
rigorous mathematical concepts and methods, to address this problem and others of 
similar nature. This collection of efforts has progressively defined a new discipline 
that can be called mathematical shape theory. 

Probably, the first milestone in the development of the theory is Kendall’s 
construction of a space of shapes, defined as a quotient of the space of disjoint 
points in R¢ by the action of translation, rotation, and scaling [40]. Kendall’s theory 
has been the starting point of a huge literature [15, 41,64] and allowed for new 
approaches for studying datasets in which the group of similitudes was a nuisance 
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Fig. 1 Examples of shapes (Taken from the MPEG-7 shape database) 


factor (for such data as human skulls, prehistoric jewelry, etc.). One can argue that, 
as a candidate for a shape space, Kendall’s model suffers from two main limitations. 
First, it relies on the representation of a shape by a finite number of labeled points, or 
landmarks. These landmarks need to have been identified on each shape, and shapes 
with different numbers of landmarks belong to different spaces. From a practical 
point of view, landmarks are most of the time manually selected, the indexation of 
large datasets being time consuming and prone to user-dependent errors. The second 
limitation is that the metric on shapes is obtained by quotienting out the standard 
Euclidean metric on point sets, using a standard “Riemannian submersion” process 
that we will discuss later in this chapter. The Euclidean metric ignores a desirable 
property of shape comparison, which states that shapes that are smooth deformations 
of one another should be considered more similar than those for which the points in 
correspondence are randomly displaced, even if the total point displacement is the 
same. 

This important issue, related to smoothness, was partially addressed by another 
important contribution to the theory, which is Bookstein’s use of the thin plate 
splines originally developed by Duchon and Meinguet [10, 16,51]. Splines interpo- 
late between landmark displacements to obtain a smooth, dense, displacement field 
(or vector field). It can be addressed with the generic point of view of reproducing 
kernel Hilbert spaces [7,77], which will also be reviewed later in this chapter. 

This work had a tremendous influence on shape analysis based on landmarks, 
in particular for medical studies. It suffers, however, from two major drawbacks. 
The first one is that the interpolated displacement can be ambiguous, with several 
points moved to the same position. This is an important limitation, since inferring 
unobserved correspondences is one of the objectives of this method. The second 
drawback, in relation with the subject of this chapter, is that the linear construction 
associated to splines fails to provide a metric structure on the nonlinear space of 
shapes. The spline deformation energy provides in fact a first-order approximation 
of a nonconstant Riemannian metric on point sets, which provides an interesting 
version of a manifold of landmarks, as introduced in [11, 36, 72]. 

After point sets, plane curves are certainly the shape representation in which 
most significant advances have been observed over the last few years. Several 
important metrics have been discussed in publications like [37,42,43,52,79-81]. 
They have been cataloged, among many other metrics, in a quasiencyclopedic effort 
by D. Mumford and P. Michor [55]. We will return to some of these metrics in 
section “Spaces of Plane Curves.” 
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Grenander’s theory of deformable templates [27] is another seminal work for 
shape spaces. In a nutshell, Grenander’s basic idea, which can be traced back to 
D’Arcy Thomson’s work on biological shapes in the beginning of last century 
[65], is to introduce suitable group actions as generative engines for visual object 
models, with the natural use of the group of diffeomorphisms for shapes. While the 
first developments in this context use linear approximations of diffeomorphisms 
[2, 28, 29], a first computational breakthrough in the nonlinear estimation of 
diffeomorphisms was provided in [12] with the introduction of flows associated 
to ordinary differential equations. This idea was further developed in a fully metric 
approach of diffeomorphisms and shape spaces, in a framework that was introduced 
in [17, 66, 67] and further developed in [8, 11,35, 36,57, 58]. The approach also led 
to important developments in medical imaging, notably via the establishment of a 
new discipline, called computational anatomy, dedicated to the study of datasets of 
anatomical shapes [30, 60, 61,78]. 


3 Mathematical Modeling and Analysis 
Some Notation 


The following notation will be used in this chapter. The Euclidean norm of vectors 
a € R¢ will be denoted using single bars and the dot product between a and b as 
a-b or explicitly as a” b, where a’ is the transpose of a. So 


lal? =a-a=a'a 


fora € R¢. 

Other norms (either Riemannian metrics or norms on infinite-dimensional 
spaces) will be denoted with double bars, generally with a subscript indicating the 
corresponding space, or relevant point in the manifold. We will use angles for the 
corresponding inner product, with the same index, so that, for a Hilbert space V, the 
notation for the inner product between V and w in V will be (v, w), with 


lvl, =v. vy. 


When / is a function that depends on a variable 1, its derivative with respect 
to tf computed at some point fp will be denoted either 0, f(to) or f; (to), depending 
on which form gives the most readable formula. Primes are never used to denote 
derivative, that is, f’ is not the derivative of f, but just another function. The 
differential at x of a function of several variables F is denoted DF (x). If F is scalar 
valued, its gradient is denoted V F(x). The divergence of a vector field v : R? > R@ 
is denoted V - v. 

If M is a differential manifold, the tangent space to M at x € M will be denoted 
T,M and its cotangent space (dual of the former) T;* M. The tangent bundle (dis- 
joint union of the tangent spaces) is denoted TM and the cotangent bundle T* M. 
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When 1 is a linear form on a vector space V (i.e., a scalar-valued linear transfor- 
mation), the natural pairing between yz and v € V will be denoted (j1| v), that is, 


(H| v) = 2). 


A Riemannian Manifold of Deformable Landmarks 


Interpolating Splines and RKHSs 

Let us start with some preliminary facts on Hilbert spaces of functions or vector 
fields and their relation with interpolating splines. A Hilbert space is a possibly 
infinite-dimensional vector space equipped with an inner product which induces 
a complete topology. Letting V be such a space, with norm and inner product, 
respectively, denoted || - ||y and (- , -)y, a linear form on V is a continuous linear 
transformation yz : V + R. The set of such transformations is called the dual space 
of V and denoted V*. An element jz in V* being continuous by definition, there 
exists a constant C such that 


We V,p(v) < Cllvllv. 


The smaller number C for which this assertion is true is called the operator norm of 
and denoted || || y*. 

Instead of ju(v) like above, the notation (| v) will be used to represent the result 
of ys applied to V. The Riesz representation theorem implies that V* is in one-to- 
one correspondence with V, so that for any yz in V*, there exists a unique element 
v= Kyw € V such that, for any w € V, 


(u|w) = (Ky, w)y; 


Ky and its inverse Ly = Kj! are called the duality operators of V. They 
provide an isometric identification between V and V*, with, in particular, ||,.||}-« = 
(| Kv) = ||Kvelly- 

Of particular interest is the case when V is a space of vector fields in d 
dimensions, that is, of functions v : R¢ > R¢ (or from Q — R¢@ where Q is an 
open subset of R“), and when the norm in V is such that the evaluation functionals 
a ® 6, belong to V* for any a,x € R¢, where 


(a @ 6,|v) = a’ v(x), ve V. (1) 


In this case, the vector field Ky (a ® 6x) is well defined and linear in a. One can 
define the matrix-valued function (y, x) > Ky(y, x) by 


Ky(y,x)a = (Ky (a ® 4y))(y); 
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K v is the kernel of the space V. In the following, we will write Ky (x, y) instead 
of Ky (x, y), with the customary abuse of notation of identifying the kernel and the 
operator that it defines. 

One can easily deduce from its definition that Ky satisfies the reproducing 


property 
Va,b € R*, (Ky(-,x)a, Ky(-,y)b)y =a" Ky(x, yb, 


which also implies the symmetry property Ky(x,y) = Ky(y,x)’. Unless 
otherwise specified, it will always be assumed that V is a space of vector fields 
that vanish at infinity, which implies the same property for the kernel (one variable 
tending to infinity and the other remaining fixed). 

A space V as considered above is called a reproducing kernel Hilbert space 
(RKHS) of vector fields. Fixing such a space, one can consider the spline inter- 
polation problem, which is to find v € V with minimal norm such that v(x;) = c;, 
where X1,...,Xy are points in R¢ and cy,...,cy are d-dimensional vectors. It is 
quite easy to prove that the solution takes the form 


N 
v(y) = > Kv, xian, (2) 
i=l 
where a,...,@y are identified by solving the dN-dimensional system 
N 
Ky Geer Sc for 7 = 122. (3) 


i=l 
Let Sy (x) (where x = (x1,...,Xy)) denote the dN by dN block matrix 
Sv (x) = (Kv Qi. Xj) jain: 


Stacking c,,...,cy and @,,...,@y in dN-dimensional column vectors ¢ and w, one 
can show that, for the optimal V, 


Ilvlj- =e" Sy (x)a = e7 S(x)y'e, (4) 


each term representing this spline deformation energy for the considered interpola- 
tion problem. 

How one uses this interpolation method now depends on how one interprets 
the vector field V. One possibility is to consider it as a displacement field, in the 
sense that a particle at position x in space is moved to position x + v(x), therefore 
involving the space transformation g’ := id + v. In this view, the interpolation 
problem can be rephrased as finding the smoothest (in the V-norm sense) full 
space interpolation of given landmark displacements. The deformation energy in 
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(4) can then be interpreted as some kind of “elastic” energy that evaluates the 
total stress involved in the transformation g’. This (with some variants, including 
allowing for some no-cost affine, or polynomial, transformations) is the framework 
of interpolation based on thin plates, or radial basis functions, as introduced in 
(3,4, 9, 10, 18], for example. As discussed in the introduction, this approach does 
not lead to a nice mathematical notion of a shape space of landmarks; moreover, in 
the presence of large displacements, the interpolated transformation g” may fail to 
be one to one and therefore to provide a well-defined dense correspondence. 

The other way to interpret V is as a velocity field, so that v(x) is the speed of a 
particle at x at a given time. The interpolation problem is then to obtain a smooth 
velocity field given the speeds c),...,cy of particles xj,...,xy. This point of 
view has the double advantage of providing a diffeomorphic displacement when 
the velocity field is integrated over time and allowing for the interpretation of the 
deformation energy as a kinetic energy, directly related to a Riemannian metric on 
the space of landmarks. 


Riemannian Structure 
Let Lmky denote the submanifold of R“ consisting of all ordered collections of 
N distinct points in R¢: 


Linky = {x = (x1,...,%~) € (R17) x; # x; if i F j}. 
The tangent space to Lmky at x can be identified to the space of all families of d- 


dimensional vectors ¢ = (c;,..., Cy), and one defines (with the same notation as in 
the previous section) the Riemannian metric on Lmky 


llellx =e" Sy(x)le. 
As already pointed out, |{¢||2 is the minimum of ||v||7, among all V in V such that 
v(x;) = c;,i = 1,..., N. This minimum is attained at 
N 
vC)= > K(-,x;)oj 
i=] 
with « = Sy(x)~!e. 


Now, given any differentiable curve ¢ + x(t) in Lmky, one can build an optimal 
time-dependent velocity field 


v6, -) = VC) 


with c = 0;x. One can then define the flow associated to this time-dependent veloc- 
ity, namely, the time-dependent diffeomorphism ¢”, such that y’(0, x) = x and 


0,p' (t,x) = v(t, p(t, X)) 
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which is, by construction, such that g(t, x;(0)) = x; (t) fori = 1,..., NV. So, this 
construction provides a diffeomorphic extrapolation of any curve in Lmky, which 
is optimal in the sense that its velocity has minimal V norms, given the induced 
constraints. The metric that has been defined on Lmky is the projection of the V 
norm via the infinitesimal action of velocity fields on Lmky,, which is defined by 


v-(X1,...,Xv) = (v(%}),..., v(Xy)). 
This concept will be extensively discussed later on in this chapter. 


Geodesic Equation 
Geodesics on Lmky are curves that locally minimize the energy, that is, they are 
curves ¢ +> x(t) such that, for any f, there exists h > 0 such that 


tth 5 
/ IeuCw) [Panel 
t 


—h 


is minimal over all possible curves in Lmky that connect x(t —) and x(t +h). The 
geodesic, or Riemannian, distance between Xo and x; is defined as the minimizer of 
the square root of the geodesic energy 


1 
. 2 
/ IiulZ yy 
0 


over all curves in Lmky that connect xo and x). 

Geodesics are characterized by a second-order equation, called the geodesic 
equation. If one denotes Gy (x) = Sy(x)7!, with coefficients 8ki),,j) for k,l = 
1,...,N andi, j = 1,...,d, the classical expression of this equation is 


N d 
. ki . 
Seat DE DE Tae inttatey = 0. 


IV=V j,j/=1 
where ee jy) are the Christoffel symbols, given by 


(ki) ! 
(Lj). W7) = 5 (Any, Suna) ote Ox)j (ki). U7) Oxe guna): 


Tr 

In these formulae, the two indices that describe the coordinates in Lmky , xx; were 

made explicit, representing the ith coordinate of the kth landmark. Solutions of this 
equation are unique as soon as x(0) and x(0) are specified. 

Equations put in this form become rapidly intractable when the number of 

landmarks becomes large. The inversion of the matrix Sy(x) or even simply its 

storage can be computationally impossible when WN gets larger than a few thousands. 
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It is much more efficient, and analytically simpler as well, to use the Hamiltonian 
form of the geodesic equation, which is (see [38]) 


0,x = Sy(x)a 


0, = —Fx(a Sy (xa) ©) 


This equation will be justified in section “General Principles,’ in which the 
optimality conditions for geodesics will be retrieved as a particular case of general 
problems in calculus of variations and optimal control. Its solution is uniquely 
defined as soon as x(0) and (0) are specified. The time-dependent collection of 
vectors t b> a(t) is called the momentum of the motion. It is related to the velocity 
c(t) = x(t) by the identity ce = Sy (x)e. 

Introducing Ky and letting Ky (x, y) denote the i, j entry of Ky(x, y), this 
geodesic equation can be rewritten in the following even more explicit form: 


N 
OX = > Ky (xe. xan, k= ly ciey lV, 

7. 7 (6) 
0;0K = -) » Vi Ky (xn, x1)oKiaj, kK=1,...,N, 


1=1i,j=1 


where V, K 7 denotes the gradient of the 7, 7 entry of Ky with respect to its first 
variable. 

The geodesic equation defines the Riemannian exponential map as follows. Fix 
xo € Lmky. The exponential map at Xo is the transformation ¢ +> exp,, (¢) defined 
over all tangent vectors c to Lmky at xo (which are identified to all families of d- 
dimensional vectors, c = (c,...,Cy)), such that CXPx, (c) is the solution at time 
t = 1 of the geodesic equation initialized at x(0) = xo and x(0) = ce. Alternatively, 
one can define the exponential chart in Hamiltonian form that will also be called the 
momentum representation in Lmky by the transformation 


My > exp, (@o), 


where expr, (a) is the solution at time | of system (6) initialized at (xo, a). 

For the metric that is considered here, one can prove that the exponential map 
at Xo (resp. the momentum representation) is defined for any vector ¢c (resp. a); 
this also implies that they both are onto, so that any landmark configuration y can 
be written as y = exp, (ao) for some wo € (R“)". The representation is not one 
to one, because geodesics may intersect, but it is so if restricted to a small-enough 
neighborhood of 0. More precisely, there exists an open subset U C Tx,Lmky over 
which exp,, is a diffeomorphism. This provides the so-called exponential chart at x 
on the manifold. 
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Fig. 2 Metric distortion for the exponential chart 


Metric Distortion and Curvature 

Exponential charts are often used for data analysis on a manifold, because they 
provide, in a neighborhood of a reference point xo, a vector-space representation 
which has no radial metric distortion, in the sense that, in the chart, the geodesic 
distance between Xo and exp,,(c) is equal to ||c||x,. The representation does distort 
the metric in the other directions. One way to measure this is by comparing (see 
Fig. 2), for given eo and ¢; with |[Collxo = |leillx) = 1, the points exp, (teo) and 
exp,, (¢(¢o + se;)). Let F(t, s) denote the last term (so that the first one is F(t, 0)). 
One can write 


dist(F'(t,s), F(t,0)) = sllas F(t, 0)llr@.o) + o(s). 


Without metric distortion, this distance would be given by st ||c; ||x, = st. However, 
it turns out that [14] 


Fe 
lO; F(t, 0) |Le¢0) = t — px(Co, we + o(t), 


where (x(Cp, ¢;) is the sectional curvature of the plane generated by cp and c; in 
Tx,Limky . So, this sectional curvature directly measures (among many other things) 
the first order of metric distortion in the manifold and is therefore an important 
indication of this distortion of the exponential charts. 

The usual explicit formula for the computation of the curvature involves the 
second derivatives of the metric tensor matrix Gy(x), which, as we have seen, 
is intractable for large values of N. In a recent work, Micheli [53] introduced an 
interesting new formula for the computation of the curvature in terms of the inverse 
tensor, Sy (Xo). 


Invariance 

The previous landmark space ignored the important facts that two shapes are 
usually considered as identical when one can be deduced from the other by a 
Euclidean transformation, which is a combination of a rotation and a translation 
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Fig. 3 Riemannian submersion (geodesics in the quotient space) 


(scale invariance is another important aspect that will not be discussed in this 
section). To take this into account, we need to “mod out” these transformations, 
that is, to consider the quotient space of Lmky by the Euclidean group. 

One can pass from the metric discussed in the previous section to a metric on the 
quotient space via the mechanism of Riemannian submersion (Fig. 3). The scheme 
is relatively simple, and we describe it and set up notation in a generic framework 
before taking the special case of the landmark manifold. So, let Q be a Riemannian 
manifold and xz : Q — M be a submersion, that is, a smooth surjection from Q 
to another manifold M such that its differential Dz has full rank everywhere. This 
implies that, form € M, the set x~!(m) is a submanifold of Q, called the fiber at 
m. If q € Q andm = z(q), the tangent space 7,Q can be decomposed into the 
direct sum of the tangent space to 2~!(m) and the space perpendicular to it. We will 
refer to the former as the space of vertical vectors at g, and denote it V,, and to the 
latter as the space of horizontal vectors, denoted ,. We therefore have 


TyQ = Vy -L Hq. 


The differential of 2 at g, Dx(q), vanishes on V, (since z is constant on 1! (m)) 
and is an isomorphism between Hy and T,, M. Let us make the abuse of notation 
of still denoting Dz(q) the restriction of Di(q) to Hy. Then, if q,q’ € x~!(m), 
the map pg’,q := Dx(q)~' o Dx(q') is an isomorphism between H,’ and H,. One 
says that a is a Riemannian submersion if and only if the maps py’,, are in fact 
isometries between H,’ and H, whenever q and q’ belong in the same fiber, that is, 
if one has, for all v’ € Hy, 
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Il Pa’.aV' lla = IIv'llq- 
Another way to phrase this property is 
m(q) = 1(q').v € Hg. V € Hg, Dx (qv = Da(q’' = |Wvllq = Iv’ Ila 


A Riemannian submersion naturally induces a Riemannian metric on M, simply 
defining, form € M andh € T,,M, 


[Alm = |Dx(q) "Allg 


for any q € x~'(m), the definition being independent of g by assumption. This is 
the Riemannian projection of the metric on Q via the Riemannian submersion z. 

Let us now return to the landmark case, and consider the action of rotations and 
translations, that is of the special Euclidean group of R¢, which is traditionally 
denoted SE(R“). The action of a transformation g € SE(R“) on a landmark 
configuration x = (x,,...,xy) € Linky is 


gx= (g(x1),..-,2(Xy)). 


We want to use a Riemannian projection to deduce a metric on the quotient space 
M = Lmky /SE(R“) from the metric that has been defined on Lmky, the surjection 
x being the projection z : Lmky — M, which assigns to a landmark configuration 
x its equivalence class, or orbit under the action of SE(R“), defined by 


[x] = {g-x,g € SE(R“)} € M. 


To make sure that M is a manifold, one needs to restrict to affinely independent 
landmark configurations, which form an open subset of Lmky and therefore let O 
be this space and restrict 2 to Q. In this context, one can show that a sufficient 
condition for x to be a Riemannian submersion is that the action of SE(R“) is 
isometric, that is, for all g € SE(IR“), the operation dg :X +> g-xis such that, for 
allu,v € TxQ, 


(Dag(x)u, Dag(x)v),., = (u, V)y. 
This property can be translated into equivalent properties on the metric. For 
translations, for example, it says that, for every x € Q andt € R%, one must 
have 


Sy(x + tT) = Sy(x) 


which is in turn equivalent to the requirement that, for all x, y,t € R¢, Ky(x + 
Tt,y+t) = Ky(x, y), so that Ky only depends on x — y. With rotations, one needs 
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diag(R)’ Sy (Rx)diag(R) = Sy(x), 
which again translates into a similar property for the kernel, namely, 
R" Ky(Rx, Ry)R = Ky(x,y). 
Here, R is an arbitrary d dimensional rotation, and diag(R) is the dN by dN block- 
diagonal matrix with R repeated N times. 


Kernels that satisfy these properties can be characterized in explicit forms. These 
kernels include all positive radial kernels, that is, all kernels taking the form 


Ky (x,y) = v(x — y/)Idpa, 


where y : [0, +00) — R is the Laplace transform of some positive measure ju, that 
1S, 


y(t) = i; e duly). 


Such functions include Gaussians, 


y(t?) = exp(—t*/20’), (7) 
Cauchy, 
(Qe? = (8) 
Se Teg? 


or Laplacian kernels, defined for any integer c > 0 by 


c 1 
yell?) = > He.) exp(-t/o) (9) 


i=1 


with p(c,1) = 2'~*(2c —1)---(e 41-1) /1!. 

One can also use non-diagonal kernels. One simple construction of such kernels 
is to start with a scalar kernel, for example, associated to a radial function y as 
above, and, for some parameter A > 0, to implicitly define Ky via the identity, valid 
for all pairs of smooth compactly supported vector fields V and w, 


T = Svc fe 
: [, v(x)" Ky (x, y)w(y)dxdy = 2 is v (Ix — yl?) (ox) wy) dxdy 


i 2 
"> a i. v (lx — yl?)(V-vQ))(V + w(y))dxdy, 
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where (V - ) denotes the divergence operator. The explicit form of the kernel can be 
deduced after a double application of the divergence theorem yielding 


Ky (x,y) = (v(r*) — Ap(r) dg — 22H (r°)(x — y)(x— y)" 


with r = |x — y|. 

Assume that one of these choices has been made for Ky, so that one can use a 
Riemannian submersion to define a metric on the quotient space Q /SE(R“). One 
of the appealing properties of this construction is that geodesics in the quotient 
space are given by (equivalent classes of) geodesics in the original space, provided 
that they are initialized with horizontal velocities. Another interesting feature is 
that the horizontality condition is very simply expressed in terms of the momenta, 
which provides another advantage of the momentum representation in Eq. (6). Take 
translations, for example. A vertical tangent vector for their action at any point 
x € M isa vector of the form (t,..., 7), where t is a d-dimensional vector repeated 
N times. A momentum, or covector, & is horizontal if and only if it vanishes when 
applied to any such vertical vector, which yields 


N 
ya = 0. (10) 
k=1 


A similar analysis for rotations yields the horizontality condition 


N 
(OK x, - xp0g ) = 0. (11) 
k=1 


These two conditions provide the d(d + 1)/2 constraints that must be imposed 
to the momentum representation on M to obtain a momentum representation on 
M/SE(R‘). 


Hamiltonian Point of View 


General Principles 
This section presents an alternate formulation in which the accent is made on 
variational principles rather than on geometric concepts. Although the results 
obtained using the Hamiltonian approach that is presented here will be partially 
redundant with the ones that were obtained using the Riemannian point of view, 
there is a genuine benefit in understanding and being able to connect the two of 
them. As will be seen below, working with the Hamiltonian formulation brings new, 
relatively simple concepts, especially when dealing with invariance and symmetries. 
It is also often the best way to handle numerical implementations. 

To lighten the conceptual burden, the presentation will remain within the 
elementary formulation that uses a state variable g and a momentum jp, rather 
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than the more general symplectic formulation. On a manifold, this implies that the 
presentation is made with variables restricted to a local chart. 

An optimal control problem in Lagrangian form is associated to a real-valued 
cost function (or Lagrangian) (q,u) t+ L(q,u) defined on QO x U, where QO is a 
manifold and U is the space of controls, and to a function (g,u) + f(q,u) € T,Q. 
The resulting variational problem consists in the minimization of 


[ L(q,u)dt (12) 


subject to the constraint g, = f(q,u) and some boundary conditions for q(¢;) and 
q(t7). The simplest situation is the classical problem in the calculus of variations for 
which f(q,u) = u, and the problem is to minimize i L(q, qr)dt. Here, [t,t] isa 
fixed finite interval. The values ¢; = 0 and¢¢ = 1 will be assumed in the following. 

The general situation in (12) can be formally addressed by introducing Lagrange 
multipliers, denoted p(t), associated to the constraint 0;¢ = f(q,u) at time f; p is 
called the costate in the optimal control setting. One then looks for critical paths of 


1 
Jala. po) = flaw) + (lar flaw) dt 


where the paths p, u, and q vary now freely as far as q(0) and q(1) remain fixed. 
The costate is here a linear form on Q, that is, an element of i, QO. 
Introduce the Hamiltonian 


Aq, p.u) = (p| f(g, u)) — L(g. u) 


for which 


1 
Jo= [ (ola) ~ Ha, p.wpat 


Writing the conditions for criticality, 6Jo/du = 6Jo/dq = 6Jo/dp = 0, directly 
leads to the Hamiltonian equation: 


dq = 0,H, 0p = —0,H, 0,H = 0. (13) 


The above derivation is only formal. A rigorous derivation in various finite- 
dimensional as well as infinite-dimensional situations is the central object of 
Pontryagin Maximum Principle (PMP) theorems which state that along a solution 
(qx, Px, Ux), one has 


A (q(t), Px (1), Ux (1) = max H(gs(t), x(t), Ww). 
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Introducing H (q, p) = max, H(q, p, u), one gets the usual Hamiltonian equation: 
ip = —d,H, oq = pH. (14) 


One can notice that, in the classical case f(q, uv) = u, H(q, p) coincides with the 
Hamiltonian obtained via the Legendre transformation in which a function u(p, q) 
is defined via the equation p = 0,L and 


(p,q) = (p| u(q, p)) — L(g, u(q, p)). 


Application to Geodesics in a Riemannian Manifold 

Let Q be a Riemannian manifold with metric at g denoted (- , -),. The computation 
of geodesics in Q can be seen as a particular case of the previous framework in at 
least two (equivalent) ways. The first one is to take 


L(x, u) = |ull;/2 and f(x, u) = u, 


which gives a variational problem in standard form. For the other choice, introduce 
the duality operator K, : fig Q — T,Q defined by 


(a| §) = (Kye, &),, 
ae ‘a Q,& € T,Q, and let, denoting the control by a, 
L(q,a) = (a| Kya) /2 and f(qg,a) = Ka. 
The Hamiltonian equation in this case yields p = a and 


0g = Kya, 


1 15 
0, = —59q((a| K,a)). o) 


This equation obviously reduces to (5) with g = x, Kga = Sy(x)a. 


Momentum Map and Conserved Quantities 

A central aspect of the Hamiltonian formulation is its ability to turn symmetries into 
conserved quantities. This directly relates to the Riemannian submersion discussed 
in section “Invariance.” 

Consider a Lie group G acting on the state variable g € Q, assuming, for the 
rest of this section and the next one, an action on the right denoted (g,¢g) — q-g. 
Notice that results obtained with a right action immediately translate to left actions, 
by transforming a left action (g,q) > g-q into the right action (g,qg) BH g7!-q.In 
fact, both right and left actions are encountered in this chapter. The standard notation 
TiaG = 6 will be used in the following to represent the Lie algebra of G. 
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By differentiation in the g variable, the action can be extended to the tangent 
bundle, with notation (g,v) > v-g for v € TQ. By duality, this induces an action 
on the costate variable through the equality (p- g| v) = ( P\v- gy). Differentiating 
again in g at g = idg gives the infinitesimal actions on the tangent and cotangent 
bundles, defined for any € € 6 = Tig, G by (E, v) > v-& and for any (&, p) > p-é 
such that (p-&|v) + (p|v-&) = 0, forallv € TO and p € T*Q. 

Now, assume that H is G-invariant, that is, H(q-g,p-g) = H(q, p) for any 
g € G, and define the momentum map (q, p) > m(q, p) € 6* by 


(m(q, p)|€) = (plq-&). (16) 


Then, one has, along a Hamiltonian trajectory, 


d,m(p,q) = 0, (17) 


that is, the momentum map is a conserved (vectorial) quantity along the Hamiltonian 
flow. This result is proved as follows. First notice that if g(¢) is a curve in G with 
g(0) = idg and g,(0) = &, then, if H is G-invariant, 


0 = 0,H(q-g.p-g) = (9gH|q-&) + (p-§| dp H). 
On the other hand, from the definitions of the actions, one has 
(d:m(q. p)|E) = 9:(p1q-&) = (Or plq-&) — (p-&| 9), 
so that if (q, p) is a Hamiltonian trajectory, 
(d,m(q, p)|&) = —(9gH|q-&) — (p-§|0,H) =0 
which gives (17). 
Notice that the momentum map has an interesting equivariance property: 
(m(q-g,p-s)l§) = (p-8l(-g)-&) 
= (p-slq-(gé)) 
= (p| (q-(gé))-¢"') 
= (pia: ((g6)g~")) 


where gé& denotes the derivative of h + gh inh ath = idg along the direction & 
and (g&)g~! the derivative of h + hg! inh ath = g along the direction g&. The 


map & +> (g&)g~! defined on G is called the adjoint representation and usually 
denoted v +> Ad,&. One therefore gets 


(m(q-g, P-8)|€) = (plg- Ade(E)) = (m(q, p)| Ade(E)) = (Adz (m(q, p))1€), 
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where Ad; is the conjugate of Ad,. Hence 


m(q-g, p-g) = Adz (m(q, p)), (18) 
that is, m is Ad*-equivariant. 


Euler-Poincaré Equation 
Consider the particular case in which Q = G and G acts on itself. In this case, 


(m(idg, p)| v) = (ply), 


so that m(idg, p) = p and one gets from Eq. (18) 
pg | = m(idg, pg') = Ad? (m(g, p)). 


Hence, along a trajectory starting from g(0) = idg of a G-invariant Hamiltonian 
H, one has (denoting p = pg ! € &* and using the fact that the momentum map 
is conserved over time) 


p(t) = pig(ey! = Ads, (m(g(), pO) 


This is the integrated version of the so-called Euler—Poincaré equation on 6* [6,50], 
d,p + adi,,(p) = 0, (20) 


where v(p) = gg”! = 0,H(idg, pg!) = 0,H (ida, p) and ad is the differential 
at location g = idg of Adg. 

A special case of this, which will be important later, is when the Hamiltonian 
corresponds to a right-invariant Riemannian metric on G. There is a large literature 
on invariant metrics on Lie groups, which can be shown to be related to important 
finite and infinite-dimensional mechanical models, including the Euler equation for 
perfect fluids. The interested reader can refer to [5, 6, 33, 34,49, 50]. 

Such a metric is characterized by an inner product (- , -), on 6 and defined by 


(v, w), =(vg', we"). (21) 
If one lets Kg be the duality operator on © so that 
(ply) = (Kop, Ve: 


the issue of finding minimizing geodesics can be rephrased as an optimal control 
problem like in the case of landmarks, with Lagrangian L(g, “) = (| K elt) /2, 


f(g.) = Kgp, and 
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Kew = (Ko(ug'))g- (22) 
The Hamiltonian equations are then directly given by (15), namely, 


Og = Kg, 


1 23 
ayy = ~5((il Ken) ve 


This equation is equivalent to the one obtained from the conservation of the 
momentum map, which is (with p = ug!) 


Og = vg, 
v= Kep, (24) 
0; = —ad* p. 


A Note on Left Actions 
Invariance with respect to left actions is handled in a symmetrical way to right 
actions. If G is acting on the left on G, define the momentum map by 


(m(p,q)| v) = (plv-q) 


which is conserved along Hamiltonian trajectories. Working out the equivariance 
property gives 


m(g- p,g-q) = Ady_im(p,q). 


When G acts on itself on the left, the Euler—Poincaré equation reads 
p(t) = Ad*(p(0)) 
or 


d,p — adi, = 0 


with p = g~'pand w(p) = g7!&. 


Application to the Group of Diffeomorphisms 
Let G C Diff(R“) be a group of smooth diffeomorphisms of R? (which, say, 
smoothly converge to the identity at infinity). Elements of the tangent space to G, 
which are derivatives of curves t +> g(t, -) where g(t, -) € G for all t, can be 
identified to vector fields x > v(x) € R¢4,x € R?. 

To define a right-invariant metric on G, introduce a Hilbert space V of vector 
fields on R@ with inner product (- , -);. Like in section “A Riemannian Manifold 
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of Deformable Landmarks,” let Ly and Ky = Le denote the duality operators 
on V, with (v, w)y = (Lyv|w) and (uw, v)y*x = (u| Kyv); Ky is furthermore 
identified with a matrix-valued kernel Ky (x, y) acting on vector fields. 

The application of the formulae derived for Hamiltonian systems and of the 
Euler—Poincaré equation will remain in the following of this section at a highly 
formal level, just computing the expression assumed in the case of diffeomorphisms 
by the general quantities introduced in the previous section. There will be no attempt 
at proving that these formulae are indeed valid in this infinite-dimensional context, 
which is out of the scope of this chapter. As an example of the difficulties that can be 
encountered, let us mention the dilemma that is involved in the mere choice of the 
group G. On the first hand, G can be chosen as a group of infinitely differentiable 
diffeomorphisms that coincide with the identity outside a compact set. This would 
provide a rather nicely behaved manifold with a Lie group structure in the sense 
of [44,45]. The problem with such a choice is that the structure would be much 
stronger than what Riemannian metrics of interest would induce and that geodesics 
would typically spring out of the group. One can, on the other hand, try to place 
the emphasis on the Riemannian and variational aspects so that the computation 
of geodesics in G, for example, remains well posed. This leads to a solution, 
introduced in Trouvé (Infinite dimensional group action and pattern recognition. 
Technical report. DMI, Ecole Normale Supérieure, unpublished, 1995) (see also 
[70]), in which G is completed in a way which depends on the metric (- , -}y, 
so that the resulting group (denote it Gy) is complete for the geodesic distance. 
This extension, however, comes with the cost of losing the nice features of infinitely 
differentiable transformations, resulting in Gy not being a Lie group, for example. 

This being acknowledged, first consider the transcription of (23) to the case 
of diffeomorphisms. This equation will involve a time-evolving diffeomorphism 
y(t, -), and a time-evolving covector, denoted z(t), which is a linear form over 
vector fields (it takes a vector field x +» v(x) and returns a number that has so far 
been denoted (j1(t)| v)). It will be useful to apply jz(t) to vector-valued functions of 
several variables, say v(x, y) defined for x, y € R?, by letting one of the variables 
fixed and considering V as a function of the other. This will be denoted by adding a 
subscript representing the effective variable, so that 


(H(t)| VOX, Y)) 


is the number, dependent of y, obtained by applying p(t) to the vector field x b> 


v(x, y). 
One first needs to identify the operator Ky in Eq. (22), defined by 


Kyu = (Ky(ug"'))g = (Ky(ug')) 09 


since right translation here coincides with composition. Now, for any vector a € R¢ 
and y € R¢, one has 
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a" (Kpu)(y) = a" (Ky(ug '))(9Q)) 
= (4 ® 8gy| Kv (ug ')) 
= (ug | Kya @ 8g) 
= (u| Kv (a @ bg) 0g) 
= (ul Ky (g(x), g(y))a)y. 


So, letting e1,..., eq denote the canonical basis of R?@, one has 


d 


d 
(Kyu)(y) = >> ef (Kew (ye: = D> (ue KL (x), 90) 2. 


i=l i=l 
where K a is the ith column of Ky. Therefore 


d 


(| Kon) =D) (ul (Hl Kp). 90))),¢1), 


i=l 
and (using the symmetry of Ky) 


d 
(Ap(HI Kox)| w) = 2° (Hl (u| D2Ky (x), 0) (9) ,€7) 


i=] 


where DoKi, is the derivative of Ky with respect to its second variable. These 
computations directly give the transcription of (23) for diffeomorphisms, namely, 


d 
0(t.y) = D> (UO Ky (Y(t. x). oC ))) ei 
i=l] 
d 
Vw: BHO] w) = — D7 (HOI (HO)! DK (Cx). P y)WV)) ,e7) ,- 


i=l 

(25) 

To transcribe Eq. (24) to diffeomorphisms, one only needs to work out the 

expressions of Ad, and ad, in this context. Recall that Ad,w was defined by 

(yw)g—'; pw being the differential of the left translation (i.e., 0;(g 0 W(t))(0) with 

w(0) = id and 0;¥(0) = w), one finds pw = Dg w, and since right translation is 
just composition, 


Adyw = (Dow) og". 


Now, since ad,w is the differential of Adyw in g (at g = id), a quick computation 
shows that 
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ad,w = Dvw— Dwv. 
So, Eq. (24) provides 


d9(t, y) = vit, elt, y)) 
v(t, x) = Ky p(t)(x) (26) 
Vw € V, (0;e| w) = —(p(t)| Dv w — Dwv) 


with the last equation being equivalent to 


(p(t)| w) = (o(0)| De '(wo g)). 


Note that jz and p in (25) and (26) are related via 4 = pg or 


(ul w) = (pl wog™). 


Solving Eq.(25) (or (26)) between times 0 and 1 provides the momentum 
representation in G, denoted 


g(1, +) = expr, .)(H(0)). 


Equivalently, the initial velocity being Ky j2(0), this is, in the exponential chart, 


pC, -) = expye,.)(Kv4(0)). 


Reduction via a Submersion 
This section, which can be put in parallel with the discussion on Riemannian 
submersions in section “Invariance,” discusses how submersions from a manifold 
Q onto another manifold M allow for the transfer of a Hamiltonian system on Q 
to a Hamiltonian system on M, given some invariance properties satisfied by the 
Hamiltonian. 

Let mz be a submersion from a manifold Q onto a manifold M so that for any 
q € QO, Day : T7Q — Tyq)M is a surjective mapping. For any gq € O, Vy = 
Dzx(q)'(0) is the previously mentioned vertical space so that V = UgegVy will 
be called the vertical bundle. In the previous Riemannian setting, a metric on TQ 
induces the definition of a horizontal space H, at any location g € Q such that 
T,Q = V4 + Hg. In the Hamiltonian approach, the horizontal space is defined in 
the cotangent space tie QO without any reference to a particular metric as the set of 
conormal covectors to the vertical space, that is, 


He = {p € TF | (piv) =0Vve Vit. (27) 


An elementary argument in linear algebra (which is left to the reader) shows that if 


one introduces the one-to-one adjoint mapping Dx*(q) : T at > Tj Q, one has 
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Hy = Dx(x(q))* (TM). In other terms, a covector p is horizontal at g if and 
only if there exists a covector a € To q)M such that Dx(x(q))*a = p. Therefore, 
H* = UgegH, can be seen as a sub-bundle of the cotangent bundle 7* Q for which 


there exists a surjective mapping 
z:H* >T*M 


defined by 7(q, p) = (1(q), (Dx*(q))""(p)). 

The main idea of this Hamiltonian (one should say symplectic or better Poisson) 
point of view is that H* is the natural image in T* Q of the dynamic space (phase 
space) T*M on M. Now, assume that a Hamiltonian Hg is given on Q. One says 
that Ho is z-reducible if there exists a Hamiltonian Hy on T* M such that 


or equivalently 
Hy (m,a@) = Ho(q, Da(q)*a) (29) 
forg € a~!(m). 

Hamiltonian trajectories in both spaces are related as follows. Assume that (g, p) 
is Hg-Hamiltonian (i.e., 0;¢ = 0pHg and 0;p = — 0,Hg) with (q(0), p(0)) € 
H*, and consider in a similar way a Hyy-Hamiltonian trajectory (m,a@) such that 
(m(0),@(0)) = z(q(0), p(O)); then for any t > 0, one has (q(t), p(t)) € H* and 


(m(t), a(t)) = (q(t), p(t)). Equivalently, one has the commutative diagram 


®o(.,.) 
W* ee W* 


[i [i (30) 


Oy (.,., 
THM SUD pe, 


where ®y and ®g are the associated Hamiltonian flows (in particular H* is @g 


invariant). To prove this fact, first, notice that from the definition of Hy, which can 
be rewritten as 


Hy(x(q),a) = Ho(q, Dx(q)*a@), 
one gets 
(p| da Hu) = (Dx(q)*p| 0,Ho) (31) 


and (8m Hu| Dx(q)&) = (0qHo|€) + (@| D?x(q)(E.0pHo)) (32) 
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(as usual, computations are assumed to be done within a chart, and the second 
derivative of z is defined according to this chart). 


Define x(t) = (m(t),a(t)), y(t) = (g(t), p(t), z = (x,y) and the 
transformation 


V2) = (Wu). W2 (2) = (m— 21g), p— Dx*(q)a). 


One needs to prove that y(z(t)) = 0. 
If Z(z) = (0vHu,—0m Hm ,0pHo,—0, Ho) is the vector field governing the 
joint Hamiltonian flows (by construction 0;z = Z(z)), one has 


DW(2)Z = Vif W@) =0. (33) 


Notice that this fact implies that Z is everywhere tangent to the set y = 0, which 
is locally a submanifold because Dy(z) has full rank, as can easily be seen. This 
implies that yy = 0 is invariant by the flow associated to Z. 

To prove (33), assume y(z) = 0 and notice that the statement is equivalent to 
(6 | Dir (2)Z@) + (DW2(2)Z(2)|bo) = 0 for any = (¢™, Gg). One has 


(6"|Dyu(2)Z(2)) = (6" |daHu — Dx(q)dp Ho) 
and 
(Dv2(2)Z(2)| £0) = (—8g Ho + Dx*(q)8m Hu | fo) — (a| D?(q) (C0, 9p H)) 
= (9m Hu| Dx(q)bo) — (8qHol 60) — («| D?x(q)(0p H. £0). 


and the result is a direct consequence of (31) and (32). 

Let us review how this concept of reduction via a submersion property gener- 
alizes the Riemannian submersion idea. When Q and M are Riemannian with M 
equipped with the projected metric, one has by construction 


(E, n)g =(Dx(Qé&  Dx(g)n)aq) (34) 


for any q € M and& € H, horizontal at q. Let Kz : T;*Q — T,Q be the duality 
operator for the metric at q (such that (p|&) = (K, DP, é),) and K,, be the same 
operator for the metric on M. From (27), one gets 


Hg = KH; . 
If one expresses (34) for€ = K,Dzx(q)*a and n = K,Dx(q)* 6 and identifies 
the terms, one gets an equivalent version of (34) in terms of the duality operators, 


namely, 


Ky, = Dx(q)K,Dx(q)", (35) 
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the invariance assumption being that the right-hand term does not depend on 
q € m~'(m). Now, the Hamiltonians associated to the metrics, respectively, 
are Ho(q,p) = (p| Kqp)/2 and Hy(m,a) = (a| Kna)/2, and it is now 
straightforward to see that the condition Hy (m,a) = Ho(q, Dx(q)*a) if x(q) = 
m, that is, condition (28), is also equivalent to (35). 


Reduction: Quotient Spaces 

A fundamental special case of the previous situation is when zr is the projection onto 
a quotient space M = Q/G, where G, is a group of symmetries, acting on Q. A 
left action is assumed in the following, a right action being handled in a symmetrical 
way. Introduce the canonical projection z : Q — M which associates the orbit 
G;-q to an element q of Q. Let us first work out conditions that ensure that a 
Hamiltonian Hg is m-reducible. One needs 


Ho(q. p) = Ho(q'. p’) 


whenever 2(q) = 2(q’), and there exists a € TrgM such that p = Da(q)*a@ and 
p’ = Dx(q’)*a. Notice that 2(qg) = x(q’) implies that there exists a g € G such 


that g’ = g-q. From the relation Dx(q')(g-&) = Dzx(q)& which derives from 
m(g-q) = 1(q), one gets 


(pl &) = (a| Dx(q)é) = (a| Dx(q'\(g-&)) = (P'l8-&) 
which implies that p’ = g-p (this condition obviously implying that they 


correspond to the same a@ if they both are horizontal). So Hg is m-reducible if and 
only if Ho is G-invariant, namely, 


Ho(g-9.g8°p) = Ho. p) (36) 


For the construction made in the previous section to be useful in practice, one 
needs to provide a simple description of the cotangent bundle to M, T*M. This 
will be done using the momentum map m, for the action of Gs, and in particular the 
set 


m,'(0) = {(q, p) € T°: VE € Gs, (pIE-q) = 0} = H". 
Given this notation, one has the identification 
m,!(0)/G, = T*M. (37) 
First, notice that the right-hand term is meaningful, since, by the equivariance of the 
momentum map, my! (0) is invariant by G,. To prove (37), recall the transformation 


xz: H* = m!(0) > T*M by x(q, p) = (m,a@) with m = (a) and p = 
Dx(q)*a. The last identity means that 


1784 A. Trouvé and L. Younes 


(a| Dx(q)v) = (pv), 


and the condition m;(qg, p) = O implies that this definition is not ambiguous, since 
Dx(q)v = O implies that v = &-q for some &, and therefore that (p|v) = 
(m;(q, p)| &) = 0. (The definition does define (a| ) for all p because Da(q) has 
full rank, since z is a submersion.) 

The next remark is that 7 induces a map [7] on the quotient space my! (0)/G;, 
defined by 


[7 (Gs - (4, p)) = 2, p). 


Again, one must make sure that the definition makes sense by proving that 

u(g-g,g-p) = (p,q), but this is an immediate consequence of the definition 

of the extended action of G; on T*Q. Finally, [7] is one to one, since as shown 

above if x(q) = z(q') = mand p = Dx*(q)a and p’ = Dx*(q’)a, then there 

exists g € Gs such that (q’, p’) = (g-q, g- p). This proves the identification (37). 
As an example, consider the reduction of the Hamiltonian 


1 
A(x,a) = 5% Sv (x) 


in the landmark case (Q = Lmky) and the invariance by the group G, = SE(R®). 
With (| &) = ee af &, the momentum map for this action is 


N 
(m,(x,@)| (A, t)) =) af (Axx + 7) 
k=1 


defined for all skew-symmetric matrix A and vector t € R“, and the conditions for 
m,(x, a) = 0 are exactly those given in (10) and (11). 

Note that condition (35) on the duality operator directly corresponds to the 
invariance conditions associated to the kernel Ky in section “Invariance.” 


Reduction: Transitive Group Action 

Consider the situation of a eft group action Gx M — M ofa group G ona manifold 
M. The important example in this chapter is when G is a group of diffeomorphisms 
and M is a set of “shapes” (for instance, M = Lmky). Assume that the action is 
transitive, that is, G-mo = M so that m : G > M defined by z(g) = g-moisa 
smooth surjection, that will be assumed to be a submersion. The situation here is on 
how to project a Hamiltonian system on G onto a reduced one on M. 

Let 


Go = {g € G| g-mo = mo} = a '(mo) 
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be the isotropy group of mo. Then condition (29) for a Hamiltonian Hg on G 
is equivalent to the invariance of Hg to the right action of Go on G, namely, 
H¢(gh, ph) = He(g, p) forh € Go. 

Although it is often more convenient to apply the reduction directly to z as 
defined above, since the structure of T* M is generally easily defined in this context, 
it is interesting to notice that this reduction also comes as an application of the 
previous construction on quotient spaces via the well-known identification [32] 
M ~ G/Go. This identity extends to cotangent spaces as above, with 


mg! (0)/Go = T*M, (38) 


where mg is the momentum map associated with Go. 

One can interpret the construction of the Riemannian metric for landmarks within 
this framework. Take M = Linky, G a group of diffeomorphisms, and mop = Xp. If 
a = (a),...,ay) € TxM™%, one can identify p = Dm*(y)a as 


N 
p= Soa ® 8x0 35 


i=l 


since for any v € T,G, 


N 
(plv) = So@ilvOo,)) 


i=1 


and Dx(g)v = (v(x0,1),---, V(Xo0,v)). Assume that a Riemannian metric is defined 
on G such that (v, w),, is associated with a duality operator Ky that can be identified 
with a reproducing kernel also denoted Ky (without assuming right invariance yet). 
With this assumption, one has 


N 
1 1 
He (x, 0) = H(y, p) = 5(PiKop) = 5) a) Kolo, 0.) 


i=l 


The invariance assumption is now clear: one needs that Ky (xo, Xo,; ) only depends 
on X = @:Xpo. This is in particular implied by the full right-invariance assumption 
discussed in section “Application to the Group of Diffeomorphisms” for which 
Kg (x07, X0,j) = Kv (x, x;), yielding in this case 


N 
1 
Hy (x, a) = 5 ot Kv (xi. xy ory 


i=1 


in the G-invariant case. As an alternative, one could, for example, also use the less 
restrictive assumption Ky(Xo;,X0,;) = Kx(x;,x;) where Ky is still a kernel, like 
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in (7), (8), or (9), in which the scale parameter o is chosen dependent of x (e.g., 
increasing as a function of |x — xo|?). 

The situation of a fully G-invariant Hamiltonian Hg can be studied in the general 
setting. Indeed, since G acts on M, one can consider the associated momentum map 
my on 7* M defined by 


(maz (m, o)| §) = (a| § +m). 


If p = Dx*(g)(a), then pg~'! = my(m, a). Indeed, 


(pg '|€) = (plég) = (a|Dx(g)ég) = (@|E-m). 


Hence, 


Hy(m,a) = He(g, p) = He(idg, pg!) = He (idg, my (m,a)). 


In the case of an invariant Riemannian metric H(idg, p) = 5(p|Kv p) — Aas 
where || || denotes the dual norm, this gives 


1 
Hy(m,a) = 5llmarn, alles - (39) 


Spaces of Plane Curves 


Introduction and Notation 

We now consider two-dimensional shapes represented by their contours and address 
the case of spaces of plane curves. Compared to the space of landmarks, two new 
issues significantly complicate the theory. The first one is that curves are infinite- 
dimensional objects, which will place us in the framework of infinite-dimensional 
Riemannian manifolds. The second one is that curves are rarely labeled, which will 
require the analysis to be invariant by a change of parameterization. 

Let us first start with a few definitions regarding plane curves. Parameterized 
plane curves can be seen as functions x : 5! — R?, where S! is the unit circle in R?. 
For simplicity, they will be assumed to be smooth (infinitely differentiable), unless 
specified otherwise. Smooth curves over the unit circle can equivalently be seen as 
infinitely differentiable 27-periodic functions with periodic derivatives defined on 
the real line. It will be convenient to use both representations in the following. 

One says that x : S!' > IR? is an immersion (or an immersed curve) if its 
first differential never vanishes (one often also says that x is a regular curve). We 
let Z denote the space of immersed curves. Immersed curves, which are easily 
characterized by their non-vanishing first derivative, are a convenient but a relatively 
imperfect representation of two-dimensional shapes, since they may include curves 
that self-intersect. A more restrictive class is the space of embedded curves, that 
contains immersed curves that coincide, in the neighborhood of any point, and 
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after a suitable rotation, with the graph of a smooth function. But because being 
embedded is a global statement about the curve, and therefore harder to handle 
than being immersed which is just local, this discussion will primarily focus on the 
space TZ. 

We let t(u) = x(u)/|x(u)| be the unit tangent at u (or x(u)) to x; v(u) be the unit 
normal, obtained by rotating t(u) of 2/2; and x (u) be the curvature, given by 


k = (0st)! v = t) v/|x,| 


where, following [55], we let 0; denote the operator 0,,/|X,]. 

A change of parameter (or reparameterization) for a curve is a smooth diffeomor- 
phism w+> w(u) of S! or, alternatively, a smooth increasing diffeomorphism of the 
real line such that for all vu € R, 


W(ut+ 27) = Wu) + 22. 


Changes of parameter act on parameterized curves on the right via 


(W,x) Fh xoy. 


A normalized arc-length parameterization of x is a change of parameter taking the 
form 


su) = 99+ = i Lydia (40) 


and L is the length of x with 


Qn 
length(x) =a |x,,(u)|di. 
0 


The scalar number sp intervening in the arc-length parameterization can be assumed 
to be between 0 and 27 without loss of generality and will be referred to as the 
offset of the parameterization. 

The quotient space of immersed curves by reparameterization is the space of 
geometric curves, denoted B. This space can in turn be quotiented out by the actions 
of rotations, translations, and scaling, which act on the left and commute with 
changes of parameter, in the sense that the result of applying a similitude and a 
change of parameters does not depend on the order with which these operations are 
performed. 

The goal in this section is to discuss shape spaces of curves obtained by putting 
a Riemannian structure on B, possibly quotiented by Euclidean transformations 
and/or scaling. But before this discussion, it will be interesting to list a few of 
the basic distances that can be defined on this set without using a Riemannian 
construction. 
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Some Simple Distances 
We here consider some simple parameterization-free distances between curves 
based on the images of the curves (the set x(R)). 

A very simple example is to use standard norms (like L? or Sobolev norms) 
computed on the difference between two curves parameterized with their normalized 
arc length. Take, for example, the L? norm, and define for two curves x and x 
parameterized with normalized arc length 


2n 
d12(x,X) = inf (/ |x(s + 59) — X(6)/'d) (41) 
SO 0 


the infimum being taken over all possible offsets as defined in Eq. (40). 

One must apply some care when defining distances like (41) which involves some 
optimization over some parameters that affect the curves. The following statement 
(the proof of which is left to the reader) is a key for this to be a valid way of building 
distances on quotient spaces. 


Lemma 1. Let M be a metric space, with distance d : (x,x') + d(x,x’). Let G be 
a group acting on M (with, say, a left action). Assume that d is G-invariant, which 
means that, for all x,x' € M and all g € G, 


d(g-x, g-x’) = d(x,x’). 
Then the distance d defined on the quotient space M/G by 


d([x], [x']) = inf, Ug -x, -x’) (42) 


is symmetric and satisfies the triangle inequality. 
Notice that, because of the G-invariance, d is also given by 


d([x], [x’']) = inf d(g-x,x’). (43) 
geG 


A sufficient condition ensuring that d is a distance (the missing property being 
d([x], [x’]) = 0 => [x] = [x’]) is that the orbits [x] = G-x are closed subsets 
of M for all x € M. The invariance condition can be placed in parallel with the 
invariance condition that arose in our discussion of the Riemannian submersion, the 
latter being an infinitesimal version of the former in the case of Riemannian metrics. 

Returning to (41), it is easy to see that a change of offset provides a group action 
on the left on curves and that the L? distance is invariant to this action. It is not 
too hard to prove that the action has closed orbits so that (41) does provide a valid 
distance in B. Since the L? distance is also invariant by the left action of rotations 
and translations, one can also define 
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2n 
d12(x, X) = inf (/ |gox(s + so) + b— X(6)P'd) ; (44) 
S00, 0 


where gg is the rotation of angle 6 and b € R’. 
A variant of this distance directly compares the derivative of the curves, which 
provides a translation-invariant representation, defining 


2n 
dys (x, %) = 7 (/ |go9sx(s + 50) — a.5(s)Pds . (45) 
SO; 0 


This distance has been introduced for curve comparison in [48], with a very efficient 
computation algorithm based on Fourier transforms. 

When a curve x is simple (i.e., without self-intersection), it can be considered 
as the boundary of a bounded set (its interior) that will be denoted Q,. A simple 
distance comparing two such curves, say x and x’, is the area of the symmetric 
difference between Q, and Q,’, that is, 


dsym(X, X’) = area(Qy U Qy’) — area(Qy N Qy). 
A more advanced notion, the Hausdorff distance, is defined by 
dy (x, x’) = inf{e > 0,x C B,(x’) and x’ C B,(x)}, 


where B,(x) is the set of points at distance less than ¢ from x (and similarly for 
B,(x’)). The same distance can be used with Q, and Qy instead of x and x’ for 
simple closed curves, the Hausdorff distance being in fact a distance between closed 
subsets of R?. 

Instead of comparing curves that are already parameterized with arc length, one 
can start with distances that are invariant by reparameterization and quotient out this 
action as described in Lemma 1. It is not easy to come up with explicit formulae for 
such invariant distances, but here is an important example. 

Start with the supremum norm between the curves, namely, 


doo(x, x’) = sup |x(u) — x(w)|, 


which is obviously invariant by changes of parameter. The distance obtained after 
reduction is called the Fréchet distance and is therefore defined by 


dp(x,x’) = inf doo(x ow,x’). 


Note that, if, for some reparameterization y, one has doo (x o w, x’) < «, then x C 
B,(x’) and x’ C B,(x). This implies the relation 


€ > dr(x,x’) > ¢ > dy(x,x’) 
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which implies dy < dr. As a consequence, d(x, x’) = 0 is only possible when 
x = x’ up to reparameterization, which completes Lemma | in ensuring that dy is 
a distance. 

Another interesting point of view that leads to parameterization-invariant dis- 
tances is to include plane curves in a suitable Hilbert space. We have already seen 
an example of this with the L* distance based on the arc-length parameterization, 
although this one required an extra one-dimensional optimization to get rid of the 
offset. An interesting alternate option (two of them, in fact) can be obtained by 
considering curves as linear forms instead of functions. 

One can first identify a curve to a measure, which is a linear form on continuous 
functions, defined by, for a curve, x and for a function f : R* > R: 


Qn 
(ux| f) = ; F(x(u))|Xu(u)|du. 


This is clearly parameterization independent, and more precisely, “x = fy if and 
only if x = y up to reparameterization or change of orientation. 

Another point of view is to identify a curve to a current [19] or, equivalently in 
this case, to a vector measure which is a linear form on vector fields. For this, simply 
define, for f : R? > R?, 


2n 
(x f) = / s(u)? f(x(u))du. 


This is also parameterization independent, with vy = vy if and only if x = y up to 
reparameterization. 

Both (signed) measures and vector measures form linear spaces, even if not all of 
them correspond to curves. Nonetheless any norm on these spaces directly induces 
a parameterization-invariant distance between curves. Hilbert norms are specially 
attracting for this purpose because of the numerical convenience of being associated 
to a dot product. One way to build such norms is to start with a Hilbert space of 
functions on R? (resp. vector fields) for which [ux (resp. vx) is continuous and then 
use the corresponding norm on the dual space [22—25, 73]. 

Start with the case of scalar functions and consider a Hilbert space W of 
functions f : R? — R such that the evaluation functionals x 1+ f(x) are 
continuous (so that W is a reproducing kernel Hilbert space of scalar functions). 
Denote by Lw : W — W* and Kw : W* — W the duality operators on 
W, similarly to what has been introduced in section “A Riemannian Manifold of 
Deformable Landmarks” with Ly and Ky, so that for f € W and p € W*, 


If lv = Lw fl S) and |lull« = (“| Kw). 


Like in section “A Riemannian Manifold of Deformable Landmarks,” Kw is a kernel 
operator, and there exists a scalar-valued function (x, y) +> Ky(x, y) such that for 
a measure [L, 
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(Kw )(x) = [ Kw (x, y)du(y). 


This implies 


lHlly* = i: Kw(x, y)du(x)du(y) 
R?xR?2 


and directly leads to a distance between curves, namely, 
d(x, x’)? = |[lx — Lx liv (46) 


Qn Qn 
= i [Kay (x(a), x(a) Ca) fo] due 
2n 
2 / Kir (x(a) X80) | EW) |d nd! 


20 
+f Kw (x'(u), x’ (u’))|x’ (w)| [x (v’) |dudu’. 
0 Jo 


The construction associated to vector measures is similar. The space W being 
this time a space of vector fields, the discussion is identical to the one holding for 
V in section “A Riemannian Manifold of Deformable Landmarks,” with a kernel 
Ky which is matrix valued. Other than this, the resulting norm in the dual space is 
formally the same, yielding 


d(x,x’)? = |lvx — vy llfp« (47) 


2n 2n 
= i) / x! Kw (x(u), x(w’))x,(u')dudu! 
0 Jo 
2n 20 
rk vl 1\\e! (4)! t 
2f i x, Kw (x(u), x (w’))x, (uw )dudu 


20 20 
+ i, / x Kw (x'(u), x (u’)) x, (u’)dudu’. 
0 0 


Riemannian Metrics on Curves 

We now pass to the specific problem of designing Riemannian metrics on spaces 
of curves. The first issue we have to deal with is that we are now handling 
infinite-dimensional manifolds, which is significantly more complex than the 
finite-dimensional space of landmarks. Since there is more than one type of infinite- 
dimensional vector spaces, there is more than one type of infinite-dimensional 
manifolds, and the one which is appropriate when dealing with spaces of infinitely 
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differentiable curves is the class of Fréchet manifolds [31]. It is not our intent, 
here, to handle the related issues with the appropriate scrutiny, the reader being 
invited to refer to [54,55] for a more rigorous presentation. We will here simply 
state intuitively plausible facts on the structures that are defined. 

The space Z of immersed curves is open in the Fréchet space C™(S', R?) of 
infinitely differentiable functions from S! to R* (in which a sequence of curves x, 
converges to x if all its derivatives converge for the supremum norm). If x € Z,a 
tangent vector &€ € Tj,Z is an element of C®(S!, R*) that can also be considered as 
a smooth vector field along x. A Riemannian metric on Z will therefore be a norm 
on this space, namely, 


E> [8 llx 


associated to an inner product (- , -), that depends on x € Z. 

We will consider norms that allow for Riemannian projections when quotienting 
out the action of changes of parameters, as well as the action of the usual 
transformation groups, SE(R7) possibly combined with scaling. Starting with 
changes of parameters, the differential of the map x + xo y simply isE BH E oy, 
which yields the first requirement 


IE ov llxoy = [NE llx (48) 


for all x € Z, € € C™(S',R?) and smooth reparameterization wy. A simple 
way to ensure parameterization invariance is to define the norm for curves that are 
parameterized with normalized arc length, simply ensuring that the norm is invariant 
by a change of offset. 

Invariance with respect to translations, rotations, and scaling, respectively, 
requires 


lEllxto = [[Ellx. 5 € R (49) 
Ilgéllex = llEllx. g € SO(R’) (50) 
AE lax = |1& IIx. A € (, +00). (51) 


A very simple norm, which satisfies (48)—(50), is the L? norm of € relative to the 
curve arc length, which is 


2n 
lgl2 = / E(w)? ula. (52) 


This norm has been studied in [54,55] and shown to provide degenerate Riemannian 
metrics in the sense that the projected Riemannian distance between any two curves 
is zero. 

Before elaborating on this fact, consider vertical vectors for the projection of 
TZ onto the space 6 of curves modulo reparameterization. They are described as 
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follows. Tangent vectors at x to the orbit of x under the action of changes of 
parameters are obtained as € = (0,(x o W))(0,u), where € b/ wWée,u) is a 
reparameterization in u which smoothly depends on «. This yields € = 0,(0, u)x,0 
w (0, u), which implies that vertical vectors € € V,, are such that all & (u) are tangent 
to x. 

Horizontal vectors at x for the metric in (52) are therefore given by vector-valued 
functions € +» &(u) that are everywhere normal to x. It follows that if [x] and 
[x’] are two equivalent classes of curves modulo reparameterization, their geodesic 
“distance” is given by 


1 Qn 
dx.x’? = int} / [ i Pvuldudt,yO,) =x, N= x. 97H <0. 
0 


(53) 
As written above, one has the following theorem. 


Theorem 1 (Mumford—Michor). The distance defined in (53) vanishes between 
any pair of smooth curves x and x’. 


A proof of this result can be found in [54,55]. It relies on the remark that one can 
grow thin protrusions (“teeth”) on a curve at a cost which is negligible compared to 
the size of the tooth. To get the basic idea underlying this result, one can understand 
how open segments can be translated at arbitrary small geodesic cost. First, consider 
a path that starts with a horizontal segment; progressively grow an isosceles triangle 
of width ¢ and height f (at time ¢) somewhere on the segment until t = 1. A quick 
computation shows that the associated geodesic length is o(e) (in fact, O(e7 Ine)). 
This implies that one can cover the horizontal segment with O(1/e) thin non- 
overlapping teeth at cost O(¢ Ine). With a similar construction and the same cost, 
one can pull up the triangles pointing downward to obtain a translated segment. The 
total cost of the operation being arbitrarily small when ¢ — 0, the geodesic distance 
between parallel segments is zero. This can in fact be extended to any pair of close 
or open curves, yielding the result stated in Theorem 1. 

Quite interestingly, small variations in the definition of the metric are sufficient 
to address this issue. Take, for example, the distance associated with 


Qn 
[2 = length(x) ; I (w)[?Beuldu, (54) 


introduced in [52,62]. Looking back at the previous “tooth example,” the length of 
a teeth being approximately 2, we see that the length term penalizes the geodesic 
energy when growing O(1/e) teeth by an extra (1/e) factor, and the total energy is 
not negligible anymore. In fact, the associated distance is not degenerate, as shown 
in [62], in which the geodesic length is proved to correspond to the total area swept 
by the time-dependent curve. 

Another way to control degeneracy is to penalize high curvature points, using, 
for example, 
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Qn 
lgI2 = / (1 + aeg(u)?)/8(w)/?Ixuldw. (55) 


This metric has been studied in [55], where it is shown (among other results) that 
the distance between distinct curves is positive. 
All the previous metrics could be put in the form 


2n 
isle = [ pxl wo? hxoldu (56) 
0 
where px > 0 is invariant by reparameterization, in the sense that 


Pxoy 0 W = px. 
More generally, one can consider metrics associated to positive symmetric linear 


operators € +> A,& which associate to a smooth vector u +> &(u) along x another 
smooth vector, u +> (A,&)(u), with the properties that 


Qn Qn 
f (n)" (Axé)|Xuldu = J (Axm)" & [Xuldu 
0 


and 


Axoy (§ 0 W) = (Ax8) 0 V. 


The geodesic equation associated to such a metric can be derived by computing 
the first variation of the geodesic energy. The computation is straightforward if one 
makes the following assumption on the variations of the operator Ay. Assume that 
there exists a bilinear operator D’ A, that takes as input two vector fields along x, 
say &(-) and y(-), and return a new vector D’ A,(&, 4)(-) such that 


Qn Qn 
af Aneag6)alialdu = f° (D'As(E 9)" Sliuld 
0 


where the derivative in the left-hand side is evaluated at ¢ = 0. With this notation, 
the geodesic equation is 


1 1 
Ay (Anke) + (85%) Aa + 59s((Anki) rt) = 5 D!AnK%) 7) 


with 0, = 0,/|X,| as above. 
This class of metrics includes the so-called Sobolev metrics [52,56] for which 
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(Axé) Edu = Yacce f |ake |" du 
wu k=0 q 


with positive coefficients a, (x), typically depending on the length of x. Let us take 
one simple example that has interesting developments: define 


2n 2n 
(Axé)! Edu = length(x)~! / |d,é|7du (58) 
0 0 


or Ay€ = —length(x)~!d?&. The metric associated to Ax is degenerate, since it 
vanishes over constants. But it provides a metric on curves modulo translations. It 
satisfies the invariance properties described above, characterized in (48), (50), and 
(51). This metric was first introduced in [79] and further studied in [68, 69, 81]. A 
direct computation shows that, in this case, 


D'Ax(&,9) = 2length(x)!d,((0,€)7 0,97) _ length(x)~!(é » Kv. 


The study of this metric is, however, much simpler than replacing the expression 
of D’ Ax into (57) would make believe. The simplification comes after the following 
transformation of the curve representation. Consider the transformation, defined 
over pairs of real-valued functions u +> (a(u), b(u)) by 


x(u) = (5 fw —b’)dit, [ abd), u € [0, 27], (59) 
2 Jo 0 
so that 
X, = (a — b’)/2 , ab). 


With the notation above, we have |x,| = (a? + b*)/2. This generate a curve in R’, 
with length 


(lJall3 + ||bI]3) . 


i 1 
length(x) = =f (a? + b’)du = a 
0 


Denoting by x = T7(a,b) the transformation in Eq.(59), one can write the 
differential of T as 


DT(a,b)(a@, B) : ure ( [a —bB)di, [va + ap)dit) 


and a direct computation yields 
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Fig. 4 An example of a 
geodesic connecting a circle 
to a star-shaped curve for the 
metric defined in (58). The 
evolving curves are 
superimposed with 
progressively reduced size to 
facilitate visualization (the 
compared curves having both 
length 1 originally) 


vllells + BI 


|| DT (a, b)(, B)||T (a,b) = 2———. 
llall + [bl 


NN 


Restricting to closed curves with unit length implies the conditions 
llallz = |[bll2 = 1 and (a, b), = 0 


which means that (a, b) forms an orthonormal two-frame in the space L*(S'), that 
is, an element of the Stiefel manifold St(L7, 2). Up to the factor two, the mapping 
T is then an isometry between St(L7, 2), equipped with its standard metric, and the 
subset of Z consisting of unit-length curves. If one furthermore makes the reduction 
of quotienting out rotations for curves, one finds that the isometry becomes with 
the Grassmannian manifold Gr(L?,2) of two-dimensional subspaces of L*. This 
identification can be exploited to obtain explicit geodesics in the considered shape 
space (see [81]). It is important to notice that the restriction to curves with unit 
length is equivalent to making the Riemannian projection on the quotient space 
modulo scalings. This is because horizontal vectors for the scale action can easily be 
shown to satisfy f(0;&)’t = 0, which, if € = x,, directly implies that 0,( |x,|*) = 
0. Therefore, length is conserved along horizontal geodesics, which justifies the 
choice of unit-length curves. Some numerical issues associated to this metric are 
studied in [68] and [69] in the simpler case in which it is applied to open curves. An 
example of geodesic obtained using this metric is provided in Fig. 4. 

A parameterized variant of this metric, applied to closed curves with unit length, 
has been proposed in [43], in the form 
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2a 


Qn 
elle = ; ((d,)€"t)*|Kuldu + c f (0) v)?|Xuldu, 


the previous metric corresponding to c = 1. When c $ 1, the unit length constraint 
is not induced by a Riemannian projection, but the metric can be studied on this 
space anyway. 

One can analyze this metric in the following way. Let x(t, u) be a time-dependent 
curve. Define A = |x,| so that 


Osx; = A7'0, (At) = 0,(logA)t + 0;t. 


Since the two terms in the sum are perpendicular, this gives 


Qn 
lel? = i, (a 10g 4)? + cl4,x/2) kyla. (60) 
0 


The first term measures the logarithmic variation of the arc length, and the second is 
the instantaneous rotation of the tangents. Interestingly, another change of variable 
akin to the one discussed for c = | can also simplify this metric in the case c = 4. 
Take, in this case, 


x=T(a.b):= ure { f ava + Paul. [ bla + aw) 
0 0 


one has this time 


Qn 
|DT (a,b). Bran = 4 (a? + p?)du 


which provides an identification of the space of open curves with unit length with an 
infinite-dimensional sphere. This identification has the important property to carry 
over to higher-dimensional curves [37]. There is, however, no “nice” representation 
for closed curves in this case. 

Notice that the two identifications that were just discussed apply to parameterized 
curves. In both cases, the geodesic distance must be optimized with respect to 
reparameterization to obtain a metric between geometric curves. 

Another important contribution to the theory of spaces of plane curves was 
made in [63], in which simple closed domains in R* are represented via the 
correspondence maps between the conformal mapping of their interior and of their 
exterior to the unit disc. This induces an almost one-to-one representation of simple 
curves by diffeomorphisms of the unit circle. In fact, this representation has to come 
modulo Mobius transformations on the circle, which are very simply accommodated 
by an invariant metric, called the Weil—Peterson metric, on such diffeomorphisms. 
The reader is referred to the cited work for more details. 
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Projecting the Action of 2D Diffeomorphisms 
At the exception of the one just mentioned, the previously discussed metrics were 
all defined based on the parameterizations of the curves. This provided reasonably 
simple definitions, exploiting in particular the invariance property of the arc length. 
Because they relied on local properties of the curves, these metrics were not able to 
penalize singularities that occur globally, like the intersection of two remote parts. 
One way to handle global constraints is to use an approach similar to the 
one that has been used to define the landmark manifold, based on the action of 
two-dimensional diffeomorphisms on curves. This will therefore be based on the 
projection paradigm discussed in section “Reduction: Transitive Group Action.” 
So, let G C Diff(R?) be a group of smooth diffeomorphisms of R* (which, say, 
smoothly converge to the identity at infinity), and let xp be a reference curve or 
template. Consider the set M = G-xo, the orbit of xo under the action of G, the 
latter being simply defined by 


(g-x)(W) = exw). 


This implies that Da(y)v = vo Xo and a horizontal covector at g € G for the 
projection takes the form 


(p| v) = (pl Vv Xo) 


for some p € Tyx,)M%*. 
Let’s make this explicit for p belonging to an important class of linear forms on 
T;M, associated to vector measures, that is, 


where j4 is a measure on the unit circle and a is a vector-valued function. The 
associated horizontal covector is then 


Qn 
(lv) = [ v(xo(u))? a(u)dye(u) (61) 


and the reduced Hamiltonian computed on this covector is (denoting as in sec- 
tion “Reduction: Transitive Group Action” Kg the duality operator on 7,G, still 
assumed to be associated to a reproducing kernel) 


1 Qn 20 
Hus) = 5 ff aw)" KyCsoluy.xol atu’ydyel data) (62) 


with x = @- Xo. As in section “Reduction: Transitive Group Action,” the invariance 
requirement boils down to Ky(xo(u), Xo(u’)) only depending on ¢-xo, with the 
simplest choice associated to a right-invariant metric on G, yielding 
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1 2n Qn 
Hw (%.p) = 5 / i au)” Ky(x(u),xw’)aw)du(w)du(u) —-«(63) 


for a fixed kernel Ky. An important fact is that measure covectors remain so during 
the evolution, the Hamiltonian (or geodesic) equations are simply written as 


20 
a,x(t,u) = Ky (x(t, u), x(t, i) a(t, a)dp(i) 
We | ae (64) 
d,a(t, uw) = — ae a! (t, u)a/ (t,i2)V K" (x(t, u), x(t, @)) dp) 
0 


ij=l 


Another interesting fact is that (64) exactly provides (6) in the case when yu is a 
weighted sum of Dirac measures. This is because these equations are, as proved 
in section “Reduction via a Submersion,” all particular instances of the Hamiltonian 
system (or geodesic equation) obtained on the acting group of diffeomorphisms, 
namely, (25). 

This was the first step downward, from diffeomorphisms to parameterized plane 
curves. It remains to discuss the additional steps, which are the reduction for the 
required invariance, by reparameterization and Euclidean transformation. 

Consider the action of reparameterization, which is a right action. The action of 
change of parameters on vector measures like in (61) is 


Qn 2n 
(p-¥18) = / a’ Foy duly) = i (ao wT Edy nu), 


where y - jz is the image of jz by w. Using this, the invariance requirement applied 
to a Hamiltonian taking the form 


1 20 20 
Huse) => ff a)” Keluu!)alu)dutu’)daw (65) 
0 0 
can be seen to reduce to the constraint that 


Kyoy (YW '(u), WW) = Kx(u,w’) 


and this property is satisfied for Kx(u, u) = Ky (x(u), x(w’)). 
The momentum map associated to changes of parameters is 


Qn 
(n(x, p)|¥) = / ati) eva and, 


so that horizontal vector measures simply are those for which a is normal to the 
curve, that is, a(u) = w(u)v(u), where & is scalar valued and v is the normal to x. 
The evolution equations then become 
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2n 
ax(t,u)= | Ky(x(t,w),x(t,))a(t,a)v(t,adin 
0 


2 20 
0,00 (t, u) = —or(t, u) > / a(t,i)v' (t,u)v/ (t,u)V, KY (x(t,u), x(t, a) dit. 
ij=ie? 
(66) 
If Ky is furthermore invariant by rotation and translation, quotienting out these 
operations results in additional conditions on a. Invariance by translation requires 


2n 
/ a(u)v(u)du = 0, 
0 


and the constraint associated to rotations is 


Qn 
/ a(u)(v(u)x(u)? — x(u)v(u) du = 0. 
0 


Extension to More General Shape Spaces 


The construction based on the Riemannian submersion from groups of diffeomor- 
phisms can be reproduced in a large variety of contexts, essentially for any class of 
objects that can be deformed by diffeomorphisms. This can be applied to provide 
metrics on space of surfaces and spaces of images, of vector fields, of measures, etc. 

Let us consider, for example, the case of images that we will take as differentiable 
functions J : R¢ — R. Define the left action of a diffeomorphism g on an image J 
to be 


g:-I=Iog. 


From this, one sees that the infinitesimal action of a vector field V on J is 
ve Tf =—v' VI. 


(This is why we assumed that the images are differentiable. For non-differentiable 
images, v- J is not a function, but a distribution, with, if o is a smooth function, 


wtp) =f 1¥-(prds. 


where V- is the divergence operator. The reader is referred to [75, 76] for the 
analysis of the inexact matching approach in the more general case of images with 
bounded variations. ) 

Fix a reference image Jp and consider the space 


M = {g- 1,9 € G}, 


Shape Spaces 1801 


the surjection being as usual 7(y) = g- Jo. Consider covectors on M that are 
associated to measures, namely, 


(lg) =f eordeon, 


where & is a real-valued function (which represents a tangent vector to M). The 
differential of z(y) = Ip 0 y~! is (letting wy = yg!) 


Da(g)v = —(VIp0g')D(g!)vog ! =-VI' vog! 


with I = @- Jo, so that the horizontal covector at g € G associated to a measure p 
is p = Dx(¢)*p defined by 


(ly =— f ween" T1400, 


where J = @- J. Starting from a Hamiltonian associated to a right-invariant metric 
on G yields the reduced Hamiltonian 


II 


1 
Hu (1, p) sf [, VI(x)" Ko(o (x), @ | (9) VI(y)dp(y)dp(x) 


II 


1 
5 ff, 9209" Kr VLA.) dpC) 
Rd JRd 
with Ky(x, y) = Ky (g(x), g(y)). The associated Hamiltonian equations are 


ae = fea VI(x)" Ky (x, y)VI(y)dp(y)dy 
0,a = V-(aKy(VIp)) 


A limitation in the image case is that two given images are very rarely connected 
by diffeomorphisms, so that working with images that are deformations of a 
reference image is a strong restriction. This issue can be addressed by extending 
the projection to a larger set than the sole group of diffeomorphisms. One can use 
a simple construction for this: call M the space of all smooth images (instead of 
just an orbit, as it was defined before), and still let G denote a group of smooth 
diffeomorphisms. Consider the surjection z : G x M — M defined by 


m(g,t)=o-l. 


(This is obviously a surjection since J = m(idga, [).) 
Letting J = J 0 g™', one has 


Dr(g,1)0,£) =-VJ  vog! +£&o@g), 
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so that the horizontal covector at (g, J) associated to a measure p on M is p = 
Dzx(g, 1)* p such that 


(1.8) = f CVITV097 +09 dp 


with J = I og7'. Thinking of a covector p € Tiy,1)(G x M)* as a pair (p, n) with 
p € T,G* and n € T; M*, one can identify (p, n) = Dx(g,1)* pas 


(ely =- f VJT vo" 'dp and (nl 8) = [ Eo ‘dp. 
Rd Rd 


If do = pdx is absolutely continuous with respect to Lebesgue’s measure, the 
second term gives 7 = po gdetDy dx. 
If one starts with a Hamiltonian on G x M for which 


1 Xr 
H((@.Do(prnd3)) = 5(Pl Kyp) +5 [n(x erDytxyy'as 


with Ky as above, the resulting reduced Hamiltonian on M is 
1 
Hu(d.pdx) => ff eyV IG)" Kv (x. VIO) edxdy 


Xr 
+ =f p(x)dx. 
2 Rd 


The corresponding evolution equations then are 


0,J = VJ' Ky(pVJ) +.Ap 
dp = V-(pKy(pVJ)). 


This is a particular instance of the theory of metamorphosis applied to images (the 
interested reader can refer to [35,71] for further developments). 


Applications to Statistics on Shape Spaces 


An important situation in which the previously discussed concepts are relevant is 
for the analysis of shape samples, that is, families x;,...,x,, in which each x; 
is a shape, possibly represented as a collection of landmarks or a plane curve (or 
another representation, like surfaces, images, etc.), and interpreted as a point in a 
manifold M.A simple and commonly used approach to analyze such samples is 
to “normalize” them using the exponential or momentum representation relative to 
a fixed template x. Each shape x; is then transformed into a tangent or cotangent 
vector, say &, € 73M so that x, = exp;(&;). 
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The problem is then reduced to the well-explored context of data analysis in 
a linear space, and how it is analyzed afterward depends on the specific problem 
at hand and is out of the scope of the present discussion. An important thing that 
one should remember is that this reduction can be accompanied with significant 
metric distortion, related to curvature as described in section “Metric Distortion and 
Curvature” (in spaces with positive curvature, the representation may even fail to 
be one to one). The approach has however proved to be a powerful analysis tool in 
several applications [20, 42], including the analysis of medical data [74]. 

This distortion being larger when the distances between the represented shapes 
and the template are large, it is natural to select the template in a way that minimizes 
these distances, the most widespread approach being to define it as a Karcher (or 
geometric) mean, that is, as a minimizer of 


U(x) = D7 di (&. xx). (67) 
k=1 


This well-posedness of this definition is also related to the curvature. The function 
U is convex and the minimum is unique if M has negative curvature [39]. Negative 
curvature is unfortunately difficult to obtain in shape spaces because the reduction 
process always increases the sectional curvature [59] (notice however that the 
representation in [63] has negative curvature, but it seems to be the only such 
example). The sectional curvature on the landmark manifold, as shown in [53], can 
be both positive and negative. As proved in [39], a sufficient condition ensuring 
the convexity of U (67) is that the diameter of the sample set (the largest geodesic 
distance between two of the points) is smaller than z/ (2s/Smax)s where Sax 1S a 
positive upper bound of the sectional curvature (U is always convex with negative 
curvature). Interestingly, in that case, the optimality condition of the Karcher mean 
is that it constitutes a sample average in the exponential representation, that is, 
X; = exp;(&;) with )°;_, &, = 0. This leads to an algorithm for the computation 
of the mean, which can be proved to converge under similar curvature conditions 
[46,47]: start with an initial guess for x and compute the exponential representation 
&, over the sample set. Compute —= a1 &;/n, replace x by exp; (—é ), and 
iterate until stabilization. A variation of this algorithm has been proposed in [21]. 
One can also mention the interesting algorithm proposed in [13] in which kernel 
regression is generalized to shape manifolds. 


4 Numerical Methods and Case Examples 


The most important numerical method on the previously discussed shape spaces 
is related to the computation of geodesics (i.e., solving the geodesic equation) 
and, most importantly in practice, to the computation of the representation in the 
exponential chart or of the momentum representation. 
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This section will focus on the latter problem (which anyway includes the first 
one as a subproblem) that will first be addressed in the simpler case of the landmark 
manifold. 

To compute exponential coordinates around some object xo, one needs to solve, 
for some target object y, the equation 


exp, (6) = y (68) 


or, if the momentum representation is more convenient, 
b 
eXp,, (@) = y. (69) 


Since these representations are defined by nonlinear evolution equations, this is a 
highly nonlinear problem, in which the function to be inverted cannot be written 
in closed form. Also, in the case of curves, the problem is infinite dimensional and 
must therefore be properly discretized. Another non-negligible issue is that, even if 
the equation has a solution (which is often the case in the discussed framework), this 
solution is not necessarily unique unless y is close enough to Xo. For this reason, it 
may be impossible to represent a generic shape dataset using only one of these 
charts, but this may be achievable for a more focused one (like shapes of fish, or 
leaves, of fixed anatomical organs). 

There are mainly two options to address the computation. The first one is to 
directly solve the equation (using zero-finding methods, like Newton’s algorithm). 
The second one is to return to the definition of geodesics as curves with minimal 
energy and to solve the variational problem of finding minimal energy paths between 
Xo and y. 


Landmark Matching via Shooting 


Let us start with the first approach. Recall that given some differentiable function 
F : R"” > R", Newton’s method to solve the equation F(z) = 0 iterates (starting 
with a good guess of the solution, zp) 


Zke+1 = 2 — DF (x) | F(z). 


This scheme can be directly applied to the solution of (69) in the landmark case, with 
F(ao) = exp, (ao) —y, since it is finite dimensional; one needs this to compute the 
differential of the momentum representation, which is only described in (6) via the 
solution of a differential equation. As a result, the differential of F', which is also 
the differential of exp, must also be computed by solving a differential equation. 
Noting that (6) takes the form 
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0,x = O(x, a) 
0,0 = R(x, a, a) (70) 


with Q linear and R quadratic in w, and that x(t) = exp, (tay), we have denoting 
I(t) = Dexpy, (tao) (71) 


a, JB = 9) O(x.a)IB + Olx, HB) a 
0;HB = 0,R(x,a,a0)JB + 2R(x,a, HB) 


in which H is an auxiliary operator that represents the variation in « (and 0, is the 
differential with respect to the first variable). Solving (70) and (72) up to time t = 1 
provides x(1) and J(1), and the Newton step is given by 


okt! — ok — F(1)"1(x(1) —y). 


Making explicit the expressions of 0; Q and 0, R is not difficult, but rather lengthy, 
and these expressions will not be provided here (the interested reader can refer 
to [1] for more details). An important limitation for the feasibility of this kind 
of approach is the cost involved in the computation of the full matrix J(¢t). With 
N landmarks in d dimensions, the size of x and a isn = Nd and the size of 
J is n*. The computation of the right-hand size of (72) requires an order of n° 
operations if one takes advantages of the special structure of the operator Q(x, -) 
(it would be n* otherwise). Even with this reduction, a computation cost which is 
cubic in the number of landmarks rapidly becomes unfeasible, and it is difficult to 
run this algorithm with, say, more than a few hundred landmarks. On the other hand, 
convergence (when it happens) can require a very small number of steps. 

Another limitation of Newton’s method is the fact that it is not guaranteed to 
converge, unless the starting point (ce with our notation) is close enough to the 
solution, in a way which is generally impossible to quantify a priori. For this reason, 
the method is often usefully complemented (and possibly replaced if the number 
of landmarks is too large) by simple gradient descent in which the minimized 
function is 


T 
F (ao) = (exp, (eo) — yy) (exp, (#0) — y). 
The first variation of F is, with the previous notation, 
T 
de F (Ho + €B)|,-o = 2 (expy, (eo) —y) J()B. 
It is natural to define gradients relative to the Riemannian metric at Xo, as defined in 


Eq. (4). When working with momenta as done here, the gradient should be identified 
using 
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magi 


Fig. 5 Example of geodesics between two landmark configurations: Jeff, trajectories (diamonds 
move onto circles), and right, resulting diffeomorphism 


BT Sy (Xo) V F(a) = 2 (exp, (#0) — y)' JU)B 
yielding 
VF (co) = 2Sy (xo)! (J(1)" (expy, (wo) — y)) . (73) 


The computation, for a given vector z, of J(t)’z can be done by solving backward 
in time the system 


3,8 = —(010(x,a))"§ — (1. R(x,a,0))"a 5 
d;a = —O(x)’& —2R(x, a)! a 

initialized with (€(1),a(1)) = (z,0), with the notation Q(x)B = Q(x, B) and 
R(x, «)B = R(x,o, B). One then has J(1)z = a(0). The proof of this statement 
derives from elementary computations on linear dynamical systems. 

This implies that the term J(1)” (expy, (ao) — y) can be computed by solving an 
ODE which has the same dimension as the geodesic Eq. (70). Notice, however, that 
(74) requires using the solution of (70) with a backward time evolution (from t = 1 
to t = 0). This implies that the solution of (70) must be first computed and stored 
with a fine enough time discretization to allow for an accurate solution of (74). This 
may cause memory issues for high-dimensional models. An example of trajectories 
and deformations estimated using this algorithm is provided in Fig. 5. 

The above discussion only addressed the computation of geodesics in landmark 
shape space without quotienting out rotations and translations. Recall that this 
operation, when done starting from a metric for which the projection on the quotient 
space is a Riemannian submersion, only requires to constrain the momentum 
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representation with a finite number of linear relations. The associated reduction 
in the number of degrees of freedom is balanced by the reduced requirement of 
connecting the reference shape to some element of the orbit of the target under 
the quotiented out group action, instead of the target itself. More explicitly, the 
equations that need to be solved to compute the momentum representation of y 
relative to Xo are 


exp, (a) — g-y = 0 


N 

Yo ao. =0 

k=l (75) 
N 

Y (eo4x8, — xo4ey) = 0. 

k=1 


where g € SE(IR“). A transformation g in this space is represented by a rotation 
part, R, and a translation part, b, and classically parameterized in the form 


R b A a) 
= exp 
0O---0 1 0---0 O 


with A skew symmetric and w € R@. System (75) therefore has Nd + d(d + 1)/2 
equations and variables and can be solved as above using Newton iterations when 
feasible or gradient descent. Since the exponential is the solution of a differential 
equation (0, exp(tU) = U exp(tu )) , optimization in A and w above can be treated 
exactly like the optimization in a. Another option is to directly use the formula 


1 
0, exp(U + é€h))._, =f exp(tU )hexp(-tU)dt. 
0 


Landmark Matching via Path Optimization 


The other option, in order to compute the momentum representation, is to solve the 
shortest path problem between Xo and y, that is, to minimize 


1 
E(x(-)) = / IE lyy dt 


with the constraints x(0) = xo and x(1) = y, using gradient descent on the space of 
all trajectories f +> x(t). Letting P(x, &) = ||&||2, one has 


1 . 
dE (x(+) + €&(-)) pcp = / (2(% i) + 3, P(x, i)"8) dt. 
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Using gradient descent requires selecting an appropriate metric on the space of 
all time-dependent objects, and interesting developments arise when selecting a 
metric for which the constraints are continuous functionals [26, 37]. Consider, as 
an example, the inner product 


Ls 
GO.) = f i ade 


that we restrict to the space of time-dependent & and 7 that vanish at t = 0 and 
t = 1. One can check that, defining 


ny (t) = i Sy (x(w))X;(u)du — [ / 0, P(x(i), (i) did u 
+(1 + of 0, P(x(u), &-(u))du, 
one has 
deE(x(-) + €&(-))uo = (§. VE(R)) 
with 


VE(x)(t) = y(t) — t9, (1). 


One can therefore use gradient descent to minimize the geodesic energy, in the form 


x" FV) = xO) — (gym (t) — tx (1). 


Computing Geodesics Between Curves 


We now discuss whether, and how, the previous methods extend to the computation 
of minimizing geodesics in Riemannian spaces of curves. We start with the metric 
associated with the projection from 2D diffeomorphisms, since it belongs to the 
same family as the one discussed with landmarks. In fact, there is a simple way to 
discretize a curve matching problem so that it boils down to a landmark matching 
problem. Assume that a reference curve xo and a target curve y are given but 
that they are only observable in discrete versions, as sequences of points le = 
(Xo,1,---,Xo,v) and y“8° = (y,,..., yy). Then, as we have remarked, Eq. (25) 
when restricted to discrete momenta of the form 


N 
w=) a @8y, 
k=1 
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boils down to Eq. (6), and one can now solve the problem of finding a solution of 
this equation that transports xo, to yg exactly as in the previous sections. 
Unfortunately, such an approach has little practical use, given that it is very 
unlikely that two discrete curves are observed such that the points that constitute 
them are exactly homologous. This means that one should not require a given xo,x 
to transform exactly into y,, but maybe to another y; or in between two of them. 
Most of the time, anyway, the curves are given with different numbers of points. 
This issue is obviously the discrete form of the parameterization invariance 
that has been discussed in section “Spaces of Plane Curves.” We know that the 
horizontality condition for parameterization invariance induces the constraint that 
ax is perpendicular to the reference curve. In this context, the problem in the 
continuum is formulated as: given Xo and y, find an initial momentum f4o which 
is horizontal at xg and such that the solution of (25) transforms the curve x into a 
deformed curve g(1, x9) which coincides with y up to a change of parameterization. 
A change of parameter being a diffeomorphism of S', it can be generated with 
an equation like (25). Roughly speaking, this change of parameter can be generated 
by momenta that are scalar functions on the unit circle. Horizontal geodesics in 
spaces of curves (still roughly speaking) are generated by momenta that are normal 
to the reference curve, which can also be represented as scalar functions on the 
unit disc. So, one needs to find two scalar functions (one for the reparameterization 
and one for the deformation) that bring the reference curve xo to the target y; the 
target being also characterized by two scalar functions (its coordinates), one sees 
that the dimensions match and that an approach based on zero finding is possible, 
at least in principle (there has been no attempt so far in the literature to solve 
the curve comparison problem in this way). The problem needs to be properly 
discretized, using, for example, the same number of points to represent xo, y, the 
reparameterization momentum, and the deformation momentum. 
One can also use a variational approach in the initial momentum, using an 
objective function like 


E(ao) = d (exp, (ao), y) (76) 


where d is a reparameterization-invariant distance, like the ones in Eqs. (46) and 
(47), which are, since they derive from Hilbert norms, well amenable to variational 
computations. The initial momentum dp can be discretized as 


N 
ao = >. aon ® Sup 


k=1 


where u1,..., Uy is a discretization of the unit disc, which, as already noticed, lead 
to geodesic equations identical to the ones considered with landmarks in (6), the 
initial “landmark positions” being xo, = Xo(ux,). This implies that the variational 
methods discussed in section “Landmark Matching via Shooting” directly apply, 
simply changing the objective function. 
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In fact, the same point of view can also be used with other metrics on 
curves, with the correct version of the exponential chart or of the momentum 
representation (whichever is more convenient). Notice that enforcing the fact that 
the initial momentum is horizontal for reparameterization is optional for these 
methods, as long as the objective function (the distance d) is parameterization 
invariant. Disregarding discretization issues, the optimal solution will always be 
horizontal, so one does not need to make an exact count of the minimal number 
of degrees of freedom, as was required by zero-finding methods. In practice, the 
computational efficiency resulting from the reduction of the number of variables can 
be counterbalanced by the additional flexibility in moving in the space of solutions 
which is offered by over-parameterized formulations, the choice between the two 
options being problem dependent. 

Finally, notice that path-minimizing methods are also available for curve match- 
ing (an approach similar to the one discussed for landmarks in the previous section 
has been proposed in [37]). 


Inexact Matching and Optimal Control Formulation 


Inexact Matching 

In many cases, requiring an exact representation of the target y in the exponential 
chart is not needed and even undesirable. In most instances, indeed, there is an 
inherent inaccuracy in the way objects are acquired. Landmarks, whether manually 
or automatically selected, are rarely well defined, and the process can lead to 
significant variability. The same holds for curves, or surfaces, which are generally 
extracted using segmentation algorithms, sometimes applied to noisy data, with 
results that cannot be assumed to be perfect. 

Formulations in which geodesics are only required to provide a good approxi- 
mation of the target then make sense and have a large range of applications. They 
are akin to the variational methods that were discussed for exact representation, in 
that they minimize an appropriate distance between the end point of a geodesic and 
the target, but they also include a penalty term on the length or the energy of the 
geodesic. In other terms, instead of minimizing d (expy,, (a0), y) like in (76), for 
example, one would minimize 


2 
E(ao) = d (exp, (ao). y) + 0° llaolls, 


(in the momentum representation, the norm is for the dual metric in the cotangent 
space at xo). If it is more convenient to use an exponential chart instead of the 
momentum representation, just minimize, over all tangent vectors & at Xo, 


E(&) = d(expy, (Eo). ¥) + 07 llEollZ, 
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Fig. 6 Two results of inexact matching with the vector-measure distance between curves as error 
term. The lower right curve (light gray) is the target. The first 11 curves provide the geodesic 
evolution 


Since this formulation only adds the term 207ap (or 207&,) to the gradient of 
the objective function that was used for exact representation (i.e., with 0” = 0), the 
methods that were described in the previous paragraphs can be adapted with minor 
changes and yield, for example, results like those provided in Fig. 6. Interestingly, 
in this context, additional methods, deriving from optimal control theory, become 
available too. 


Optimal Control Formulation 
Let us first return to the general principles discussed in section “General Principles” 
and consider an optimal control problem with an additional end-point cost EF: 


1 
minimize [ L(q,u)dt + E(q(1)) subject to g = f(g, u) and q(0) fixed. 
0 


Notice that here g(1) is free, but this situation is handled quite similarly to the 
one with fixed g(1). Introduce 


Je(q, pu) = Jog. p.u) + E(q()) 


1 
= i (L(q.u) + (pl4e — f(q,w)) at + E(q()). 


The only change in the analysis arises when working out the variation in g which 
now gives the extra end-point condition 


pi) + DE(q(1)) = 0, (77) 


which come in addition to the previously obtained (13). 

The conservation of the momentum map can be extended to this case when a 
group G acts on Q and the Hamiltonian H is G-invariant. If, in addition, E is also 
G-invariant, one deduces from E(qg) = E(q) for all g the fact that (DE(q)| &g) = 
0 for all & € G which is exactly m(q, DE(q)) = 0 where the momentum map m is 
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defined in (16). Therefore, (77) implies that m(g(1), p(1)) = 0, which, combined 
with the conservation of momentum, implies that 


m(q(t), p(t) = 0. 


for any ¢ € [0, 1]. Thus, the momentum map is not only invariant but vanishes along 
solutions of the optimal control problem. In the context of the reduction discussed 
in section “Reduction: Transitive Group Action,” this says that the momentum 
associated to a solution is horizontal. 


Gradient w.r.t. the Control 
One can compute the variations with respect to u of 


1 
es / L(q.u)dt + E(q()) 
0 


subject to the constraint ¢g = f(g, u) and q(0) fixed. 
Taking the variation with respect to this constraint yields 


0:6q = 0g f6q + 0, f du. 


Introduce the semigroup P;; solution of 0; Ps; = dg f Ps with Ps; = id, so that 


t 
ie i pCR aere 
0 


One can write 


1 t 
6C = i ((@201f Ply f bts) + (0,L| su(t)) dt 
0 0 


1 
+(De@a)) i} P.a(Buf bids). 


Interverting integrals in s and ¢ yields 
1 1 
8C = | (aL — @,fI PO] Su(s))ds = [at Bu(s)yds 78) 
0 0 
with 


1 
p(s) = -([ PY (OgL):dt + Pe, DE(a(\))) 
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II 


which is characterized by p(1) + DE(q(1)) = 0 and 0,p = d,L — 0, f*p 
—d,H(q, p, u). The last two conditions are precisely J¢/5q = 0 for 


1 
p= i ((plé) — Hq, p.w)dt + EQ) 


as above. Since Gg; = f(q,u) is 6Jz/ép = 0, one gets from (78) that 6C/du = 
bJ¢ /bu for 6J¢/bp = dJg/6g = 0. 


Application to the Landmark Case 
In the landmark case u = aw, q = x, L(x,a) = a Sy(x)a/2, and x = f(x, «@) 
Sy (x)a, so that 0,,H (x, p,«) = Sy(x)(p — a) and 


II 


1 
6C = (a — p, da(s)),ds 

0 
The gradient of C is therefore particularly simple to compute if one chooses along 
the path the natural metric given on the @’s by the matrix Sy(x) (cf. section “A 
Riemannian Manifold of Deformable Landmarks’). This gives the updating rule 
(see [22]): &"F! = a" —At(a—p"), gq *' = f(q"*',a"*'), where p” is computed 
by the backward integration of the ode p’ = —d,H(q", p",o") with end-point 
condition p”(1) + E(qg”"(1)) = 0. 


5 Conclusion 


Even if it would be impossible to provide a comprehensive description of every 
method that has been devised in this domain, this chapter provides an introduction 
to many of the mathematical constructions of spaces of shapes. The combined 
description of the Riemannian and of the Hamiltonian point of views, which are 
complementary, should help the reader to a more thorough understanding of the 
range of available methods, whether they were described in this chapter or elsewhere 
in the literature. The described numerical methods are basic components that can 
also be found in most of the contributions that were not directly addressed here. 

Mathematical shape analysis remains a domain of intensive research, with 
open problems arising both for fundamental aspects (e.g., with building spaces 
of three-dimensional shapes) and for numerical issues and their connections with 
applications. It is however likely that the concepts introduced here will remain 
relevant and serve as foundations for future work. 
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Abstract 


The concept of a shape space is linked both to concepts from geometry and from 
physics. On one hand, a path-based viscous flow approach leads to Riemannian 
distances between shapes, where shapes are boundaries of objects that mainly 
behave like fluids. On the other hand, a state-based elasticity approach induces a 
(by construction) non-Riemannian dissimilarity measure between shapes, which 
is given by the stored elastic energy of deformations matching the corresponding 
objects. The two approaches are both based on variational principles. They are 
analyzed with regard to different applications, and a detailed comparison is given. 
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1 Introduction 


The analysis of shapes as elements in a frequently infinite-dimensional space of 
shapes has attracted increasing attention over the last decade. There are pioneering 
contributions in the theoretical foundation of shape space as a Riemannian manifold 
as well as path-breaking applications to quantitative shape comparison, shape 
recognition, and shape statistics. The aim of this chapter is to adopt a primarily 
physical perspective on the space of shapes and to relate this to the prevailing 
geometric perspective. Indeed, we here consider shapes given as boundary contours 
of volumetric objects, which consist either of a viscous fluid or an elastic solid. 

In the first case, shapes are transformed into each other via viscous transport of 
fluid material, and the flow naturally generates a connecting path in the space of 
shapes. The viscous dissipation rate — the rate at which energy is converted into heat 
due to friction — can be defined as a metric on an associated Riemannian manifold. 
Hence, via the computation of the shortest transport paths, one defines a distance 
measure between shapes. 

In the second case, shapes are transformed via elastic deformations, where the 
associated elastic energy only depends on the final state of the deformation and not 
on the path along which the deformation is generated. The minimal elastic energy 
required to deform an object into another one can be considered as a dissimilarity 
measure between the corresponding shapes. 

In what follows, we discuss and extensively compare the path-based and the 
state-based approach. As applications of the elastic shape model, we consider shape 
averages and a principal component analysis of shapes. The viscous flow model is 
used to exemplarily cluster 2D and 3D shapes and to construct a flow-type nonlinear 
interpolation scheme. Furthermore, we show how to approximate the viscous, path- 
based approach with a time-discrete sequence of state-based variational problems. 


2 Background 


The structure of shape spaces and statistical analyses of shapes have been examined 
in various settings, and applications range from the computation of priors for 
segmentation [16, 17,43] and shape classification [25, 44,48, 50] to the construction 
of standardized anatomical atlases [14, 37,66]. Among all existing approaches, a 
number of different concepts of a shape are employed, including landmark vectors 
[16, 39], planar curves [41,52, 84], surfaces in R3 [24, 25, 40], boundary contours 
of objects [31, 44, 67], multiphase objects [83], as well as the morphologies of 
images [22]. 

The analysis of a shape space is typically based on a notion of a distance 
or dissimilarity measure d(-,-) between shapes [10, 31, 50, 51, 54, 67], whose 
definition frequently takes a variational form. This distance can be used to define 
an average [26, 67] or a median [4,28] S of given shapes S),...,S, according 
to S = argming )7} d(S,S;)? for p = 1 and p = 2, respectively (cf. 


i=l 
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section “Elastic Shape Averaging”). Likewise, shape variations can be obtained 
by a principal component analysis (PCA, cf. section “Elasticity-Based PCA”) or a 
more general covariance analysis in a way which is consistent with the dissimilarity 
measure between shapes [11, 16, 26,68]. From the conceptional point of view, one 
can distinguish two types of these dissimilarities or distance measures which may 
be characterized as rather state based or path based, respectively. While the first 
approach is independent of the notion of paths of shapes, the latter distance defini- 
tion requires the computation of an optimal, connecting path in shape space. In some 
cases, both concepts coincide: The Euclidean distance between two points, e.g., can 
equivalently be interpreted in a state-based manner as the norm of the difference 
vector or as the length of the shortest connecting path (we shall provide a physical 
interpretation for each case in section “Recalling the Finite-Dimensional Case’’). 

The notion of a shape space was already introduced by Kendall in 1984 [39], who 
considers shapes as k-tuples of points in R“, endowed with the quotient metric of 
IR‘ with respect to similarity transforms. Often, however, a shape space is just 
modeled as a linear vector space which is not invariant with respect to shift or 
rotation a priori. In the simplest case, such a shape space is made up of vectors of 
landmark positions, and distances between shapes can be evaluated in a state-based 
manner as the Euclidean norm of their difference. Chen and Parent [12] investigated 
averages of 2D contours already in 1989. Cootes et al. perform a PCA on training 
shapes with consistently placed landmarks to obtain priors for edge-based image 
segmentation [16]. Hafner et al. use a PCA of position vectors covering the proximal 
tibia to reconstruct the tibia surface just from six dominant modes [35]. Perperidis 
et al. automatically assign consistent landmarks to training shapes by a nonrigid 
registration as a preprocessing step for a PCA of the cardiac anatomy [63]. S6hn 
et al. compute dominant eigenmodes of landmark displacement on human organs, 
also using registration for preprocessing [73]. 

As an infinite-dimensional vector space, the Lebesgue-space L* has served 
as shape space, where again shape alignment is a necessary preprocessing step. 
Leventon et al. identify shapes with their signed distance functions and impose the 
Hilbert space structure of L* on them to compute an average and dominant modes 
of variation [43]. Tsai et al. apply the same technique to 3D prostate images [79]. 
Dambreville et al. also compute shape priors, but using characteristic instead of 
signed distance functions [19]. 

A more sophisticated state-based shape space is obtained by considering shapes 
as subsets of an ambient space with a metric d(-, -) and endowing them with the 
Hausdorff distance 


dy(S\,S2) = max{ sup inf d(x, y), sup inf d(x, y)} 


xES, VES2 yes, *£S02 


between any two shapes S;, S). Charpiat et al. employ smooth approximations of 
the Hausdorff distance based on a comparison of the signed distance functions of 
shapes [10]. For a given set of shapes, the gradient of the shape distance functional 
at the average shape is regarded as shape variation of the average and used to 
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analyze its dominant modes of variation [11]. Frame indifference is mimicked by an 
inner product that weights rotations, shifts, scalings, and the orthogonal complement 
to these transformations differently. Charpiat et al. also consider gradient flow 
morphing from one shape onto another one which can be regarded as a means to 
obtain meaningful paths even in shape spaces with state-based distance measures. 

An isometrically invariant distance measure between shapes (or more general 
metric spaces) that is also not based on connecting paths is provided by the Gromov— 
Hausdorff distance, which can be defined variationally as 


1 
dgy(S},S.) = — inf su ds, (x1,x2) —d : : 
Gu(S}, S2) Sie oe S(X1,X2) — ds,(V1, Y2)| 
W251 wy )=x; 


where ds,(-, -) is a distance measure between points in S;. The Gromov—Hausdorff 
distance represents a global, supremum-type measure of the lack of isometry 
between two shapes. Memoli and Sapiro use this distance for clustering shapes 
described by point clouds, and they discuss efficient numerical algorithms to com- 
pute Gromov—Hausdorff distances based on a robust notion of intrinsic distances 
ds(-, +) on the shapes [50]. Bronstein et al. incorporate the Gromov—Hausdorff 
distance concept in various classification and modeling approaches in geometry 
processing [6]. Memoli investigates the relation between the Gromov—Hausdorff 
distance and the Hausdorff distance under the action of Euclidean isometries as 
well as L-type variants of the Gromov—Hausdorff distance [49]. 

In [46], Manay et al. define shape distances via integral invariants of shapes and 
demonstrate the robustness of this approach with respect to noise. 

Another distance or dissimilarity measure which also measures the lack of 
isometry between shapes can be obtained by interpreting shapes as boundaries of 
physical objects and measuring the (possibly nonlinear) deformation energy of an 
elastic matching deformation ¢ between two objects [36, 67]. Since, by the axiom 
of elasticity, this energy solely depends on the original and the final configuration 
of the deformed object but not on the deformation path, the elastic dissimilarity 
measure can clearly be classified as state based (as will be detailed in section “State- 
Based, Path-Independent Elastic Setup’’). This physical approach comes along with 
a natural linearization of shapes via boundary stresses to perform a covariance 
analysis [68] and will be presented in section “Elasticity-Based Shape Space.” 
Pennec et al. define a nonlinear elastic energy as the integral over the ambient space 
of an energy density that depends on the logarithm of the Cauchy—Green strain 
tensor Dé! Dé [61,62], which induces a symmetric state-based distance. 

Typical path-based shape spaces have the structure of a Riemannian manifold. 
Here, the strength of a shape variation is measured by a Riemannian metric, and the 
square root of the Riemannian metric evaluated on the temporal shape variation is 
integrated along a path of shapes to yield the path length. The length of the shortest 
path between two shapes represents their geodesic distance d(-, -). Averages are 
obtained via the Fréchet mean [30], which was further analyzed by Karcher [38]. 
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There is also a natural linear representation of shapes in the tangent space at the 
Fréchet mean via the logarithmic map, which enables a PCA. 

A Riemannian shape space which might still be regarded as rather state than 
path oriented is given by the space of polygonal medial axis representations, where 
each shape is described by a polygonal lattice and spheres around each vertex [87]: 
Here, the Lie group structure of the medial representation space can be exploited to 
approximate the Fréchet mean as exponential map of the average of the logarithmic 
maps of the input. Fletcher et al. perform a PCA on these log-maps to obtain the 
dominant geometric variations of kidney shapes [26] and brain ventricles [27]. 
Fuchs and Scherzer use the PCA on log-maps to obtain the covariance of medial 
representations, and they use a covariance-based Mahalanobis distance to impose 
a new metric on the shape manifold. This metric is employed to obtain priors for 
edge-based image segmentation [32, 33]. 

Kilian et al. compute and extrapolate geodesics between triangulated surfaces of 
fixed mesh topology, using isometry invariant Riemannian metrics that measure the 
local distortion of the grid [40]. Eckstein et al. employ different metrics in combi- 
nation with a smooth approximation to the Hausdorff distance to perform gradient 
flows for shape matching [24]. Liu et al. use a discrete exterior calculus approach 
on simplicial complexes to compute geodesics and geodesic distances in the space 
of triangulated shapes, in particular taking care of higher genus surfaces [45]. 

An infinite-dimensional Riemannian shape space has been developed for planar 
curves. Klassen et al. propose to use as a Riemannian metric the L?-metric on 
variations of the direction or curvature functions of arc length-parameterized curves. 
They implement a shooting method to find geodesics [41], while Schmidt and 
Cremers present an alternative variational approach [70]. Srivastava et al. assign 
different weights to the L?-metric on stretching and on bending variations and 
obtain an elastic model of curves [75]. Michor and Mumford examine Riemannian 
metrics on the manifold of smooth regular curves [51]. They show the standard L?- 
metric in tangent space, leading to arbitrarily short geodesics and hence employ a 
curvature-weighted L?-metric instead. Yezzi and Mennucci resolved the problem 
taking into account the conformal factor in the metric [84]. Sundaramoorthi et al. 
use Sobolev metrics in the tangent space of planar curves to perform gradient 
flows for image segmentation via active contours [76]. Michor et al. discuss a 
specific metric on planar curves, for which geodesics can be described explicitly 
[52]. In particular, they demonstrate that the sectional curvature on the underlying 
shape space is bounded from below by zero, which points out a close relation to 
conjugate points in shape space and thus to only locally the shortest geodesics. 
Finally, Younes considers a left-invariant Riemannian distance between planar 
curves by identifying shapes with elements of a Lie group acting on one reference 
shape [85]. 

When warping objects bounded by shapes in IR¢, a shape tube in IR¢*! is formed. 
Delfour and Zolésio [20] rigorously develop the notion of a Courant metric in this 
context. A further generalization to classes of non-smooth shapes and the derivation 
of the Euler-Lagrange equations for a geodesic in terms of a shortest shape tube is 
investigated by Zolésio in [88]. 
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Dupuis et al. [23] and Miller et al. [53,54] define the distance between shapes 
based on a flow formulation in the embedding space. They exploit the fact that in 
case of sufficient Sobolev regularity for the motion field v on the whole surrounding 
domain 2, the induced flow consists of a family of diffeomorphisms. This regularity 
is ensured by a functional 1 rf gq Lv-vdx dt, where L is a higher-order elliptic 
operator [76, 85]. Geometrically, /, g Lv-vdx is the underlying Riemannian metric, 
and we will discuss related, path-based concepts in section “Path-Based, Viscous 
Riemannian Setup.” Under sufficient smoothness assumptions, Beg et al. derive 
the Euler-Lagrange equations for the diffeomorphic flow field [3]. To compute 
geodesics between hypersurfaces in the flow of diffeomorphism framework, a 
penalty functional measures the distance between the transported initial shape and 
the given end shape. Vaillant and Glaunés [80] identify hypersurfaces with naturally 
associated two forms and used the Hilbert space structures on the space of these 
forms to define a mismatch functional. The case of planar curves is investigated 
under the same perspective by Glaunés et al. in [34]. To enable the statistical analysis 
of shape structures, parallel transport along geodesics is proposed by Younes et al. 
[86] as the suitable tool to transfer structural information from subject-dependent 
shape representations to a single-template shape. 

In most applications, shapes represent boundary contours of physical objects. 
Fletcher and Whitaker adopt this viewpoint to develop a model for geodesics in 
shape space which avoids overfolding [29]. Fuchs et al. [31] propose a Riemannian 
metric on a space of shape contours, motivated by linearized elasticity. This metric 
can be interpreted as the rate of physical dissipation during the deformation of a 
viscous liquid object [82,83] and will be elaborated in section “Viscous Fluid-Based 
Shape Space.” 

Finally, a shape space is sometimes understood as a manifold, learned from 
training shapes, and embedded in a higher-dimensional (often linear) space. Many 
related approaches are based on kernel density estimation in feature space. Here, the 
manifold is described by a probability distribution in the embedding space, which is 
computed by mapping points of the embedding space into a higher-dimensional fea- 
ture space and assuming a Gaussian distribution there. In general, points in feature 
space have no exact preimage in shape space, so that approximate preimages have 
to be obtained via a variational formulation [64]. Cremers et al. use this technique 
to obtain 2D silhouettes of 3D objects as priors for image segmentation [17]. Rathi 
et al. provide a comparison between kernel PCA, local linear embedding (LLE), 
and kernel LLE (kernel PCA only on the nearest neighbors) [65]. Thorstensen 
et al. approximate the shape manifold using weighted Karcher means of the nearest 
neighbor shapes obtained by diffusion maps [77]. 


3 Mathematical Modeling and Analysis 
Recalling the Finite-Dimensional Case 


At first, let us investigate distances and their relation to concepts from physics 
in the simple case of Euclidian space. In Euclidean space, the shortest paths are 
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F= C(Xp - x1) e 
Xo F=2uv 


xy Xx; 


Fig. 1 The force F of an elastic spring between x; and x2 is proportional to (x2 — x1), as well 
as the force F of a dashpot which is extended from x, to x2 within time | at constant velocity 


v. The spring energy reads W = f Fdx = 3C lx. — x, II5 and the dashpot dissipation Diss = 
Jf F-vdt = 2p |x. — x13 


straight lines, and they are unique, so that the distance computation involves only 
the states of the two end points: The geodesic distance between any two points 
X1,X2 € IR@ is given by the norm of the difference, ||x. — x;||2, which implies 
the equivalence of the state-based and the path-based perspective. A corresponding 
physical view might be the following. Considering that — by Hooke’s law — the 
stored elastic energy of an elastic spring extended from x, to x2 is given by 
W= $C \|x2 — x] II5 for the spring constant C, the distance can be interpreted in a 
state-based manner as the square root of the elastic spring energy (Fig. 1). Likewise, 
from a path-based point of view, the minimum dissipated energy of a dashpot which 
is extended from x to x2 at constant speed within the fixed time interval [0, 1] reads 
Diss = 1 2 IIvIl3 dt = 2 ||xo — x1 3. where 2, is the dashpot parameter and the 
velocity is given by v = x2 — x. Using this physical interpretation, we can express, 
for instance, the arithmetic mean x = 1 3~’_, x; = argming )77_, ||xi — ||} of 
a given set of points X1,...,X) € R¢ either as the minimizer of the total elastic 
deformation energy in a system, where the average x is connected to each x; by 
elastic springs or as the minimizer of the total viscous dissipation when extending 
dashpots from x; to x. 

Before we investigate the same concepts on more general Riemannian manifolds, 
let us briefly recall some basic notation. A Riemannian manifold is a set M that 
is locally diffeomorphic to Euclidean space. Given a smooth path x(t) € M, 
t € [0,1], we can define its derivative x(t) at time ¢ as a tangent vector to M 
at x(t). The vector space of all such tangent vectors makes up the tangent space 
TM, and it is equipped with the metric gy ()(-, -) as the inner product. The 
length of a path x(t) € M,t € [0,1], is defined as i, Sx (X(t), X(t) dt, and 
locally the shortest paths are denoted geodesics. They can be shown to minimize 
i, Sx(t) (X(t), X(t)) dt [21, Lemma 2.3]. Let us emphasize that a general geodesic 
is only locally the shortest curve. In particular, there might be multiple geodesics 
of different lengths connecting the same end points. The geodesic distance between 
two points is the length of the shortest connecting path. Finally, for a given x € M, 
there is a bijection exp, : 7, M — M of a neighborhood of 0 € 7,M into a 
neighborhood of x € M that assigns to each tangent vector v € 7,.M the end point 
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Fig. 2 The logarithmic map 
assigns each point x; on the 
manifold M a vector in the 
tangent space TM, which 
may be seen as a linear 
representative 


logy x 


of the geodesic emanating from x with initial velocity v and running over the time 
interval [0, 1] [42, Theorem 1.6.12] or [74, Chap. 9, Theorem 14]. 

We can now define the (possibly nonunique, cf. Sect.5) mean x of a num- 
ber of n points x;,...,X, © M in analogy to the Euclidian case as x = 
argming )~)_, d(x, )*, where d(-, -) is the Riemannian distance on M. This 
average is uniquely defined as long as the geodesics involved in the distance 
computation are unique, and it has been investigated in differential geometry by 
Karcher [38]. Furthermore, on a Riemannian manifold M, the inverse exponential 
map log, = exp,! provides a method to obtain representatives log,.(x;) € TyM of 
given input points x; € M in the (linear) vector space 7}.M (Fig. 2). On these, we 
can perform a PCA, which is by definition a linear statistical tool. 

In a Riemannian space M, the path-based approach can immediately be applied 
by exploiting the Riemannian structure, and i, &x1)(X(t), X(£)) dt can be consid- 
ered as the energy dissipation spent to move a point from x(0) to x(1) along a 
geodesic. The logarithms log,.(x;) in this model correspond to the initial velocities 
of the transport process leading from x to x;. When applying the state-based elastic 
model in M, however, there is no mechanically motivated notion of paths and thus 
also no logarithmic map. Only if we suppose that the Riemannian structure of the 
space M is not induced by changes in the inner structure of our objects, the physical 
model based on elastic springs still coincides with the viscous model: We consider 
elastic springs stretched on the surface M and connecting the points x and x; with 
a stored energy 5Cd (x, xj). Then, as before in the Euclidian case, a state-based 
average x of input points x),..., x, can be defined. Furthermore, interpreting spring 
forces acting on x and pointing toward x; as linear representatives of the input 
points x;, one can run a PCA on these forces as well. However, for any reasonable 
(even finite-dimensional) model of shape space, objects are not rigid, and the inner 
relation between points as subunits (such as the vertex points of polygonal shapes) 
essentially defines the Riemannian (and thus the path-based) structure of the space 
M: The rate of dissipation along a path in shape space depends on the interaction 
of object points. Physically, the corresponding point interaction energy is converted 
into thermal energy via friction. This dissipation depends significantly on the path 
in shape space traversed from one shape to the other. In contrast, when applying 
the state-based approach to the same shape space, we directly compare the inner 
relations between the subunits, i.e., we have no history of these relations. This 
comparison can be quantified based on a stored (elastic) interaction energy which is 
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then a quantitative measure of the dissimilarity of the two objects but in general no 
metric distance. 


Path-Based Viscous Dissipation Versus State-Based Elastic 
Deformation for Nonrigid Objects 


In the following, we will especially consider two different physically motivated 
perspectives on a shape space of nonrigid volumetric objects in more detail. In the 
first case, we will adopt a path-based view, motivated by the theory of viscous fluids, 
while the second, state-based approach will be motivated by elasticity. 

We will regard shapes S as boundaries S = 0O of domains O C R¢ which 
will be interpreted as physical objects. The resulting shape space structure depends 
on the particular type of physical objects O: An interpretation of O as a blob of a 
viscous fluid will yield an actually Riemannian, path-based shape space, while the 
interpretation as an elastic solid results in a state-based perspective, which will turn 
out to be non-Riemannian by construction. 


Path-Based, Viscous Riemannian Setup 
Shapes will be modeled as the boundary contour of a physical object that is made of 
a viscous fluid. The object might be surrounded by a different fluid (e.g., with much 
lower viscosity and compression modulus), nevertheless, without any restriction 
we will assume void outside the object in the derivation of our model. Here, 
viscosity describes the internal resistance in a fluid and is a macroscopic measure 
of the friction between fluid particles, e.g., the viscosity of honey is significantly 
larger than that of water. The friction is described in terms of the stress tensor 
oO = (0j;)ij=1,..d, Whose entries describe a force per area element. By definition, 
0;; is the force component along the ith coordinate direction acting on the area 
element with a normal pointing in the jth coordinate direction. Hence, the diagonal 
entries of the stress tensor o refer to normal stresses, e.g., due to compression, 
and the off-diagonal entries represent tangential (shear) stresses. The Cauchy stress 
law states that due to the preservation of angular momentum, the stress tensor o is 
symmetric [13]. 

In a Newtonian fluid, the stress tensor is assumed to depend linearly on the 
gradient Dv := (% 


Ox; 
the stress vanishes. A rotational component of the local motion is generated by the 
antisymmetric part 5 (Dv - (Dv)") of the velocity gradient, and it has the local 
rotation axis V x v and local angular velocity |V x v| [78]. Thus, as rotations 
are rigid body motions, the stress only depends on the symmetric part €[v] := 
5 (Dv + (Dv)") of the velocity gradient. For an isotropic Newtonian fluid, we get 
oi; = Abij dO, (elke +2 (€[v]);;, or in matrix notation o = Atr (€[v]) 1+2pe[v], 
where 1 is the identity matrix. The parameter A is denoted Lamé’s first coefficient. 
The local rate of viscous dissipation — the rate at which mechanical energy is locally 
converted into heat due to friction — can now be computed as 


) 7 of the velocity v. In case of a rigid body motion, 
ij=l.... 
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diss[v] = * (ety + ptr (e[vp) . (1) 


This is in direct correspondence to the mechanical definition of the stress tensor 
o as the first variation of the local dissipation rate with respect to the velocity 
gradient, i.e, 0 = dp,diss. Indeed, by a straightforward computation, we obtain 
6(pv,, diss = A tre[v] 6;; +2 (€lv]);; = oi; . Here, tr (€[v]?) measures the averaged 
local change of length and (tre[v])? the local change of volume induced by the 
transport. Obviously, divv = tr(€[v]) = 0 characterizes an incompressible fluid. 

Now, let us consider a path (Ot) e001] of objects connecting O(0) with O(1) 
and generated by a time-continuous deformation. If each point x € O(t) of the 
object O(t) at time ¢ € [0, 1] moves in an Eulerian framework at the velocity v(t, x) 
(x = v(t, x)), so that the total deformation of O(0) into O(t) can be obtained by 
integrating the velocity field v in time, then the accumulated global dissipation of 
the motion field v in the time interval [0, 1] takes the form 


1 
diss (0), OM)<oul = fff _ tissb var. (2) 


This is the same concept as employed by Dupuis et al. [23] and Miller et al. [53] in 
their pioneering diffeomorphism approach. They minimize a dissipation functional 
under the simplifying assumption that the material behaves equally viscous inside 
and outside the object. Also, diss[v] = 4 (tre[v])? + ptr(e[v?*) is replaced by 
a higher-order quadratic form Lv-v which plays the role of the local rate of 
dissipation in a multipolar fluid model [57]. Multipolar fluids are characterized by 
the fact that the stresses depend on higher spatial derivatives of the velocity. If the 
quadratic form associated with L acts only on e[v] and is symmetric, then rigid body 
motion invariance is incorporated in the multipolar fluid model (cf. section “Viscous 
Fluid-Based Shape Space”). In contrast to this approach, we here measure the 
rate of dissipation differently inside and outside the object and rely on classical 
(monopolar) material laws from fluid mechanics. 

On this physical background, we will now derive a Riemannian structure on the 
space of shapes S in an admissible class of shapes S. The associated metric Gs 
on the (infinite-dimensional) manifold S is in abstract terms a bilinear mapping 
that assigns each element S ¢€ S an inner product on variations 6S of S (cf. 
section “Recalling the Finite-Dimensional Case” above). The associated length of 
a tangent vector 6S is given by ||6S|| = Gs(dS,6S). Furthermore, as we have 
already seen above, the length of a differentiable curve S : [0,1] — S is then 


defined by L[S] = i |S(t)|| dt = i Gs) (S(t), S(t) dt, where S(¢) is the 
temporal variation of S at time ¢. The Riemannian distance between two shapes S4 
and Sz on S is given as the minimal length taken over all curves with S(O) = Sy 
and S(1) = Sz or equivalently (cf. section “Recalling the Finite-Dimensional Case” 
above) as the length of a minimizer of the functional i Gsit) (S (t),S (t)) dt. For 
shapes S ¢€ §S, an infinitesimal variation 5S of a shape S = 0O is associated with 
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a transport field v : O —> R¢. This transport field is obviously not unique. Indeed, 
given any vector field w on O with w(x) € T,S for all x € S = dO (where 
T,S denotes the (d — 1)-dimensional tangent space to S at x), the transport field 
v + w is another possible representation of the shape variation 6S. Let us denote 
by V(éS) the affine space of all these representations. As a geometric condition 
for v € V(6S), we obtain v(x) -n[S](x) = 5S(x)-n[S](x) for all x € S, where 
n[S](x) € IR@ denotes the outer normal to S C R@ in x € S. Given all possible 
representations, we are interested in the optimal transport, i.e., the transport leading 
to the least dissipation. Thus, using definition (1) of the local dissipation rate, we 
finally define the metric Gs (6S, 6S) as the minimal dissipation rate on motion fields 
v which are consistent with the variation of the shape 6S, 


ew ' ae A 2 2 
Gs(6S, 6S) := saan, [| asst dx = cain, | , (tre[v])~ + tr (e[v}*) dx. 
(3) 


Let us remark that we distinguish explicitly between the metric g(v,v) := 
Jo diss[v] dx on motion fields and the metric Gs(SS,5S) on shape variations. 
Finally, integration in time leads to the total dissipation (2) to be invested in the 
transport along a path (S(t)),¢j9,1; in the shape space S. This implies the following 
definition of a time-continuous geodesic path in shape: 


Definition 1 (Geodesic path). Given two shapes S4 and Sg in a shape space S, a 
geodesic path between S4 and Sz is a curve (S(t)), e101, C S with S(O) = S4 and 
S(1) = Sg which is a local solution of 


min Diss [(v(t), O(t)),e0.)] 
v(thEV(S(t)) 


among all differentiable paths in S. 


The Riemannian distance between two shapes S4 and Sz induced by this definition 
is given by the length of the shortest (geodesic) path S(t) between the two shapes, 
1.€., 


Aicon(Sa,SB) = L[(S()):e0,] - 


Figure 3 shows two different paths between the same pair of shapes, one of them 
being a (numerically approximated) geodesic. Note that the chosen dissipation 
model combines the control of infinitesimal length changes via tr (c [vP). and the 
control of compression via tr (e[v]). Figure 4 evaluates the impact of these two 
terms on the shapes along a geodesic path. 


State-Based, Path-Independent Elastic Setup 
Now, objects bounded by a shape contour S are no longer composed of a viscous 
fluid but are considered to be elastic solids. To describe object deformations, we 
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Fig. 3 A geodesic (top, path length L = 0.2225 and total dissipation Diss = 0.0497) and a 
non-geodesic path (bottom, L = 0.2886, Diss = 0.0880) between an A and a B. The intermediate 
shapes of the bottom row are obtained via linear interpolation between the signed distance functions 
of the end shapes. The local dissipation rate is color coded as HD 
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Fig. 4 Two geodesic paths between dumbbell shapes varying in the size of the ends. In the top 
example, the ratio A/j between the dissipation parameters is 0.01 (leading to rather independent 
compression and expansion of the ends since the associated change of volume implies relatively 
low dissipation), and 100 in the bottom row (now mass is actually transported from one end to the 
other). The underlying texture on the objects is aligned to the transport direction, and the absolute 
value of the velocity v is color coded as 


aim for an elastic energy which is not restricted to small displacements and which 
is consistent with the first principles. Alongside the shape space modeling, we will 
recall some background from elasticity. For details, we refer to the comprehensive 
introductions in the books by Ciarlet [15] and Marsden and Hughes [47]. 

For two objects O4 and Og with shapes Sy = 0O4 andSg = dOz, we assume a 
deformation @ to be defined on O 4 and constrained by the assumption @(S4) = Sz. 
For practical reasons, one might consider O, to be embedded in a very soft elastic 
material occupying S2 \ O4 for some computational domain £2. There is an elastic 
energy Weetoml@, O4] associated with the deformation @ : 2 > R?. By definition, 
elastic means that this energy solely depends on the state and not on the path along 
which the deformation proceeds in time. More precisely, for so-called hyper-elastic 
materials, Waeramlg@, O.4] is the integral of an energy density W depending solely on 
the Jacobian D¢ of the deformation 4, i.e., 


Variational Methods in Shape Analysis 1831 


Wawel, O4] = 1 W(D6) dx. (4) 


This elastic energy is considered as a dissimilarity measure between the shapes 
S,4 and Sg. As a fundamental requirement, one postulates the invariance of the 
deformation energy with respect to rigid body motions, Weom[Q o@ + b,S4] = 
WeroomP, Sa] for any orthogonal matrix Q e€ SO(d) and translation vector 
b € RR@ (the axiom of frame indifference in continuum mechanics). From this, 
one deduces that the energy density only depends on the right Cauchy—Green 
deformation tensor Dé'D@. Hence, there is a function W : R“4 —> R such 
that the energy density W satisfies W(F) = W(F'F) for all F € R¢@“. The 
Cauchy-—Green deformation tensor geometrically represents the metric measuring 
the deformed length in the undeformed reference configuration. For an isotropic 
material and for d = 3, the energy density W can be further rewritten as a function 
WU 1,12, 13) solely depending on the principal invariants of the Cauchy—Green 
tensor, namely, J; = tr(D¢é™D@), controlling the local average change of length; 
I, = tr(cof(Dé™D¢)) (cofF := det F F~"), reflecting the local average change 
of area; and J; = det(D¢'D¢), which controls the local change of volume. For a 
detailed discussion, we refer to [15,78]. We shall furthermore assume that the energy 
density is polyconvex [18], i.e., a convex function of D¢, cofD¢, and det Dg, and 
that isometries, i.e., deformations with Dé™(x)D¢(x) = 1, are local minimizers 
with W(D¢) = W(1) = 0 [15]. Typical energy densities in this class are of the 
form 


A PB 4 
W(h, b, bb) = al? + aly + FU3) (5) 


for a,,d. > O and a convex function  : [0,0o0) — R with (1/3) > oo for 
I; 0 and J3 — oo. In nonlinear elasticity, such material laws have been proposed 
by Ogden [58], and for p = gq = 2 (the case considered in our computations), 
we obtain the Mooney-Rivlin model [15]. The built-in penalization of volume 


shrinkage, i.e., Wh, In, Ih) sie oo, enables us to control local injectivity 
(cf. [2]). 

Incorporation of such a nonlinear elastic energy allows to describe large defor- 
mations with strong material and geometric nonlinearities, which cannot be treated 
by a linear elastic approach (cf. Hong et al. [36]). Furthermore, it balances in an 
intrinsic way expansion and collapse of the elastic objects and hence frees us from 
imposing artificial boundary conditions or constraints. 

As in the previous section, the local force per area, induced by the deformation, 
is described at a point d(x) € @(Q) by the Cauchy stress tensor o. It is related 


to the first Piola—Kirchhoff stress tensor 0 = W (Dd) := _ |rF=p¢, which 
measures the force density in the undeformed reference configuration, by o = 


o ogcofD®¢. 
Based on these concepts from nonlinear elasticity, we can now define a dissimi- 
larity measure on shapes 
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Fig. 5 Example of elastic dissimilarities between different shapes. The arrows indicate the 
direction of the deformation, the color coding represents the local deformation energy density 
(in the reference as well as the deformed state) 


datas (SA , Sz) — 


Vv Weetorm [d, Oa] . (6) 


min 
$,9(Sa)=SpB 


Figure 5 shows some applications of this measure. Obviously, the elastic energy 
is in general not symmetric so that das(S4,SpB) 4 das(SB,S4). Indeed, by 
construction, du.(-, -) does not impose a metric structure on the space of shapes 
(we refer to section “Conceptual Differences Between the Path- and State-Based 
Dissimilarity Measures” for a detailed discussion). Nevertheless, it can be applied 
to develop physically sound statistical tools for shapes such as shape averaging and 
a PCA on shapes, as outlined below in section “Elasticity-Based Shape Space.” 

Let us make a brief remark on the mathematical relation between the two 
different concepts of elasticity and viscous fluids. If we assume the Hessian of 
the energy density W at the identity to be given by Wrr(1)(G,G) = A(trG)* + 
Ftr ((G + G")’) (which can be realized in (5) for a particular choice of a, a2, and 
I’, depending on the exponents p and q), then by the ansatz ¢(x) = x + tv(x) and 
a second-order Taylor expansion, we obtain 


2. 
W(D¢d) = WA) + tWr()(Dy) + 5 Warr L)(Dv. Dy) + O(r°) 
=04047 (Few? EE ctr ((Dv 4 vy"))) + O(73). 


In effect, the Hessian of the nonlinear elastic energy leads to the energy density in 
linearized, isotropic elasticity 


W'(Du) = ; (tre[u])? + ptr (€[u}’) (8) 


for displacements u with d(x) = x + u(x). This energy density, acting on 
displacements u, formally coincides with the local dissipation rate diss[v], acting 
on velocity fields v, in the viscous flow approach. 
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Finally, let us deal with the hard constraint ¢(S4) = Sg, which is often 
inadequate in applications. Due to local shape fluctuations or noise in the shape 
acquisition, the shape S, frequently contains details that are not present in Sg 
and vice versa. These defects would imply high energies in a strict 1-1 matching 
approach. Hence, we have to relax the constraint and introduce some penalty 
functional. Here, we either measure the symmetric difference of the input shapes 
S,4 and the pullback ~'(Sg) of the shape Sg given by 


F[Sa.9. Sp] = HO (SAAG'(Sz)) « (9) 
where AAB = A\ BU B \ A, or alternatively the volume mismatch 
F[S4,, Sp] = vol (O,AG | (Oz) . (10) 


Conceptual Differences Between the Path- and State-Based 
Dissimilarity Measures 

The concept of the state-based, elastic approach to dissimilarity measurement 
between shapes differs significantly from the path-based viscous flow approach. In 
the elastic setup, the axiom of elasticity implies that the energy at the deformed 
configuration Sg = @(S,4) is independent of the path from shape S,4 to shape 
Sg along which the deformation is generated in time. Hence, there is no notion 
of the shortest paths if we consider a purely elastic shape model, and different from 
a path-based approach, there might not even exist an intermediate shape Sc with 
Aas (SA,SB) = dawe(SA,Sc) + dea (Sc, SB). 

Unlike in the elasticity model, in the Newtonian model of viscous fluids, the rate 
of dissipation and the induced stresses solely depend on the gradient of the motion 
field v. Even though the dissipation functional (2) looks like the deformation energy 
from linearized elasticity as outlined above, the underlying physics is only related 
in the sense that an infinitesimal displacement in the fluid leads to stresses caused 
by viscous friction, and these stresses are immediately absorbed via dissipation. 

Surely, every (path-based) Riemannian space is metrizable (and in that sense 
state-based), and for many sufficiently regular (state-based) metric spaces, we 
can devise a corresponding (path-based) Riemannian metric. However, from our 
mechanical perspective, the conceptual difference between the path-based, viscous 
and the state-based elastic approach is striking. In the path-based approach, the 
structure of the space is too complicated for a closed formula of the geodesic 
distance, so that the actual computation of a path is required. In the state-based 
approach, there is either no underlying path (i.e., no S(¢);ej0,1] such that for any 
0<h <t <tr <I, we have d(S(t), S(h)) = d (S(t), Sh) +d (Sh), S(H))), 
or the shape space structure is simple enough to allow for a closed formula of the 
geodesic distance as in Euclidean space. 

Mathematically, the path-based nature of the viscous flow approach and the fact 
that an inversion of the motion field v + —yv leads to a path from shape Sz to Sy in 
shape space with the same dissipation and length, i.e., 
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Fig. 6 Left: viscosity-based (time-discrete) geodesics between the shapes at the corners (the 
shapes are taken from [31]). The triangle inequality holds. Right: elastic dissimilarities 
deas(*,*) = WAY = SWaeetom between the same shapes, where the arrows point from the 
reference to the deformed configuration. The triangle inequality does not hold 


VW = 0.24311 VW = 0.20460 
bad —_ — 


Fig. 7 The state-based elastic dissimilarity measure d,,, is not symmetric (as opposed to the 
path-based, viscous distance dyjisous): In this example, it costs much more energy to drag out the 
protrusion than to push it in. The color coding represents the local deformation energy density in 
the reference and the deformed configuration 


Diss [(v(0), O(t))re0,)] = Diss [(-v(1 —t),OU - t))rep0,1)] 


ensure that the associated distance dyis.,, is actually a metric. In particular, the 
symmetry condition Ayiccous(S4,SB) = Ayiscou(SB,S4) and the triangle inequality 
rviscous(SAs SC) SZ Aviscous(SAs SB) + Ariscous(SB,Sc) hold. As we have already seen, 
the symmetry condition does not hold for the elastic dissimilarity measure. Also, the 
triangle inequality cannot be expected to hold. Indeed, if a deformation ¢4,, maps 
O4 onto Og and a deformation ¢g.¢ maps Og onto Oc, then d4c := bp,c 9 b4.B 
deforms O, onto Oc. However, based on our elastic model, Og is considered to 
be stress free when applying the deformation $3 c (although it is actually obtained 
as the image of object O,4 under the deformation ¢4,,). Hence, the “history” of 
the deformation ¢4,, is lost when measuring the energy of ¢g,c. In addition, the 
energy density is highly nonlinear. As a consequence, in general, we cannot expect 
daa (SA, SC) < das (SA, SB) + dassi(SB, Sc). Indeed, Fig. 6 gives an example where 
the triangle inequality holds in the viscous, path-based and fails in the elastic, 
state-based approach. Furthermore, Fig. 7 depicts another example for the lack of 
symmetry already apparent in Fig.5 with a particularly pronounced mechanical 
difference of the two dissimilarity measures. 
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4 Numerical Methods and Case Examples 
Elasticity-Based Shape Space 


In this section, we will perform a statistical analysis on shapes up to the second 
moment, i.e., we will consider shape averaging and a principal component analysis 
on shapes as two exemplary applications of the state-based elastic shape space. 


Elastic Shape Averaging 

As usual, we consider objects © as open sets in IR@ with the object shape given 
as S := 0O. Given n sufficiently regular shapes S; = 00;,i = 1,...,n, we are 
interested in an average shape which reflects the geometric characteristics of the 
input shapes in a physically intuitive manner. Suppose S = dO C R¢ denotes a 
candidate for this unknown shape. As it is characteristic for the elastic approach, 
the similarity of the input shapes S; to S is measured by taking into account 
optimal elastic deformations ¢; : O; > IR¢ with ¢;(S;) = S. The elastic energy 
Wiaerorm| Pi , O; | of these deformations has the interpretation of a dissimilarity measure 
(cf. section “State-Based, Path-Independent Elastic Setup”), so that we obtain a 
natural definition of an average shape as the minimizer of the sum of these terms 
(cf. Sect. 2). 


Definition 2 (Elastic shape average). Given shapes S;,...,S, in some shape 
space S, the elastic shape average S is the minimizer of 


Yo daw(S;,S => _ inf Weeoml$i s Oi]. 
i=l ja] PUOIPR! Oi (S)=S 

If the input objects O; have Lipschitz boundary and the integrand of the deformation 
energy Wetom{i,O;] = So, W(D¢;) dx is polyconvex and bounded below by 
C,||D¢@; ||? — Cz for p > d, Cy, Cy > 0, the existence of a Hélder-continuous elastic 
shape average and deformations ¢; € W'?(O;) which realize the above infimum is 
guaranteed [81]. 

An example of a shape average is provided in Fig. 8. Obviously, the process of 
shape averaging is a constrained variational problem in which we simultaneously 
have to minimize over n deformations ¢; and the unknown shape S under the n 
constraints ¢;(S;) = S. 

The necessary conditions for a set of minimizing deformations are the cor- 
responding Euler-Lagrange equations. As usual, inner variations of one of the 
deformations lead to the classical system of PDEs divW-(Dd@;) = 0 for every 
deformation ¢; on O; \ S;, meaning a divergence-free, equilibrized stress field (cf. 
section “State-Based, Path-Independent Elastic Setup”). Furthermore, the coupling 
between the deformations via the constraints (#;(S;) = S);-, __, allows to derive 
a stress balance relation on S: Consistent variation of all deformations ¢; and the 
average S by some displacement u : O — R¢ via (1 + Su) o g; and (1 + du)(S) 
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Fig. 8 Elastic shape average (bottom right) of five human silhouettes. For the computation, all 
shapes have actually been described as phase fields, and the elastic deformations are extended 
outside the input objects O; (cf. section “Shapes Described via Phase Fields”). The objects O; 
are depicted along with their deformations ¢; (acting on a checkerboard) and the distribution of 
local length change FllDoi || and volume change det(D¢;) (range [0.97, 1.03] color coded as 


ms ) 


results in the optimality condition 4 yet Wieerorm [Cl + bu) 0 ;, O;] | s<90 = 9, 
which after integration by parts leads to )¥_, Js, W.r (Doi) (uogi) - v[Si] da[S)] = 
0 for the outer normal v[S;] to S;. We have here exploited divW -(Dd¢;) = 0 on 
O; \ S;. Now, we consider displacements u with local support and let this support 
collapse at some point x on S. This yields the pointwise condition 


n 


0= > (o;"v[S;] da[S;]) (6, '(x)) and thus 0 = Soils p(x) (11) 


i=1 i=l 


for x € S, where we have used the relation 
(o;"v[S;] da[S;]) (67 '(x)) = (oi v[S] da[S}) (x) 


between the first Piola—Kirchhoff stress o/'=W-(Dd¢;) and Cauchy stress 
0;= (0;"(cofD¢;)~') o ¢;'. Hence, the shape average can be interpreted as that 
stable shape at which the boundary stresses of all deformed input shapes balance 
each other (Fig. 9). Obviously, there is a straightforward generalization involving 
jumps of normal stresses on interior interfaces in case of multicomponent objects. 
In order to ensure a certain regularity of the average shape S, in addition to 
the sum of deformation energies in Definition 2, one can consider a further energy 
contribution which acts as a prior on S in the variational approach. In the exemplary 
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Fig. 9 Sketch of the pointwise stress balance relation on the averaged shape 
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Fig. 10 Average of 18 hand silhouettes (Taken from [16]) 
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Fig. 11 Five segmented kidneys and their average (right). For the first two input kidneys, the 
distribution of A ID¢; ||, i ||cof(D¢;)||, and det(D@;) is shown on sagittal cross sections (the 


range [0.85, 1.15] is color coded as HMB). While the first kidney is dilated toward the 
average, the second is compressed 
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Fig. 12 Twenty-four given foot shapes (Courtesy of Adidas), textured with the distance to the 
surface of the average foot (bottom right). Values range from 6 mm inside the average foot to 6mm 
outside, color coded as 


computations shown (Figs. 10-12), the (d — 1)-dimensional Hausdorff measure 
LIS] = H4~!(S) has been employed as regularization. 


Elasticity-Based PCA 

As already explained in section “Recalling the Finite-Dimensional Case,” a principal 
component analysis (PCA) is a linear statistical tool which decomposes a vector 
space into the direct sum of orthogonal subspaces. These subspaces are ordered 
according to the strength of variation which occurs along each subspace within a 
random set of sample vectors. We would like to interpret a given set of input shapes 
S|,...,S, as such a random sample and perform a corresponding PCA; however, 
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due to the linearity of a PCA, we first have to identify linear representatives for each 
shape on which a PCA can then be performed. For a Riemannian shape space, we 
have outlined in section “Recalling the Finite-Dimensional Case” that such linear 
representatives are given by the logarithmic map of the input shapes, but we have 
also learned in section “Conceptual Differences Between the Path- and State-Based 
Dissimilarity Measures” that a state-based elastic shape space is incompatible with 
a Riemannian structure. 

To prepare the definition of appropriate linear representatives of shapes in an 
elastic shape space, let us briefly review the physical concept of boundary stresses. 
By the Cauchy stress principle, each deformation ¢; : Ox — O is characterized by 
pointwise boundary stresses on S = dO in the deformed configuration. The stress 
at some point x on S is given by the application of the Cauchy stress tensor ox to 
the outer normal v on S. The resulting stress 0; v is a force density acting on a local 
surface element of S. The shape S is in an equilibrium configuration if the opposite 
force is applied as an external surface load (cf. Fig.9). Otherwise, by the axiom 
of elasticity, releasing the object O, the elastic body will snap back to the original 
reference configuration O,. Let us assume the relation between the energetically 
favorable deformation and its induced stresses to be one to one, so that the average 
shape S can be described in terms of the input shape S; and the boundary stress 
oxVv, and we write S = S;[o;,v]. Upon scaling the stress with a weight t € [0, 1], we 
obtain a one-parameter family of shapes S(t) = S;[toxv], connecting S; = S(0) 
with S = S(1). Thus, we can regard oxv as a representative of shape S, in the 
linear space of vector fields on S. 

Physically, it is more intuitive to identify a displacement u,; instead of the 
normal stress o;,v as the representative of an input shape S;. Hence, let us study 
how the average shape S varies if we increase the impact of a particular input 
shape S; for some k € {1,...,}. For this purpose, we apply the Cauchy 
stress 0; Vv to the average shape S, scaled with a small constant 6. This additional 
boundary stress 60, v may be seen as a first Piola—Kirchhoff stress acting on the 
(reference) configuration S. The elastic response is given by a correspondingly 
scaled displacement u, : O — IR¢. Here, to properly incorporate the nonlinear 
nature of the second moment analysis, O should be interpreted as the compound 
object which is composed of all deformed and thus prestressed input objects ¢; (O;). 
This interpretation is reflected by the elastic material law employed to compute the 
displacements u,;. In detail, uz is obtained as the minimizer of the free mechanical 
energy 


1 n 
E4[B.1] =~) Wenal (+ 81) 0 41.0) = 8 f ony ud (12) 
Ss 


i=1 


under the constraints Jo u,ydx = O and jax x up, dx = O of zero average 
translation and rotation. These displacements uz, are considered as representatives 
of the variation of the average shape S with respect to the input shape S;, on which 
a PCA will be performed. 
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As long as F + W(F) is not quadratic in F’, uz still solves a nonlinear elastic 
problem. The advantage of this nonlinear variational formulation is that it is of 
the same type as the one for shape averaging, and it encodes in a natural way 
the compound elasticity configuration of the averaged shape domain O. However, 
for the linearization of shape variations, we are actually only interested in the 
displacements éu, for small 5. Therefore, we consider the limit of the Euler— 
Lagrange equations for 6 — 0 and, after a little algebra, obtain u; as the solution of 
the linearized elasticity problem 


div (Ce[u]) =0inO, Ce[ulv = oKv onS (13) 


for the symmetrized displacement gradient ¢€[u] = 5(Du + Du’) under the 
constraints o udx = 0 and f ox x udx = 0, where the in general inhomogeneous 
and anisotropic elasticity tensor C reads 


n 


a! 1 T) 6 ¢7! 
C => (app Phe DHD0" Q - 


i=1 


Next, for a PCA on the linearized shape variations uz, we select a suitable 
inner product (metric) g(u,i) on displacements u,i : O — IR¢. Note that g 
induces a metric g(ov, dv) := g(u,u) on the associated boundary stresses so that 
instead of analyzing the u;,, the covariance analysis can equivalently be performed 
directly on the boundary stresses 0)v,...,0,V, Which we originally derived as 
linear shape representatives. Indeed, the solvability condition f, o div(CVu) dx = 
i s CVuv da[S] is fulfilled, and thus the solution wu; for given boundary stress 
oxv = CVuv is uniquely determined up to a linearized rigid body motion (ie., 
an affine displacement with skew-symmetric matrix representation), which is fixed 
by the conditions of zero mean displacement and angular momentum for u. Then, 
due to the linearity of the operator ov +> u, the metric g is bilinear and symmetric 
as well, and its positive definiteness follows from the positive definiteness of g and 
the injectivity of the map ov +> u. 

We consider two different inner products on displacements u : O > R¢: 


* The L*-product. Given two square integrable displacements u, i, we define 


g(u, uv) := / u-udx. 
oO 


This product weights local displacements equally on the whole object O. 
* The Hessian of the energy as inner product. Different from the L?-metric, we 
now measure displacement gradients in a nonhomogeneous way. We define 


g(u, iu) := [cet : €[u] dx 
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for displacements u, u with square integrable gradients. Hence, the contribution 
to the inner product is larger in areas of the compound object which are in a 
significantly stressed configuration. 


Given an inner product, we can define the covariance operator Cov by 
1 n 
Covu := — D> gZ(u, UK) UK 
n 
k=1 


(note that the stresses o;.v and thus also the displacements u; have zero mean due 
to (11)). Obviously, Cov is symmetric positive definite on span(u1,..., Un). Hence, 
we can diagonalize Cov on this finite-dimensional space and obtain a set of g- 
orthonormal eigenfunctions w;, : O > IR@ and eigenvalues A, > 0 with Covw; = 
Ax Wk. These eigenfunctions can be considered as the principal modes of variation of 
the average object O and hence of the average shape S, given the 7 sample shapes 
S},...,Sn. Their eigenvalues encode the variation strength. The diagonalization of 
Cov can be performed by diagonalizing the symmetric matrix 1 (g(ui,u iD) — 
OAO', where A = diag(A,,A2,...) and O is orthogonal. The eigenfunctions are 
then obtained as wz = Tr ini OjKuj- 

Being displacements on O, the modes of variation wz can easily be visualized 
via a scalar modulation dw, for varying 6 (cf. the vizualization in Figs. 16-18 or the 
red lines in Figs. 13 and 15). If an amplified visualization of the modes is required, 
it is preferable to depict displacements wk which are defined as minimizers of the 
nonlinear variational energy 1 Dia Waetom| (+ w) 09, O;]—8* fg CVwiv -wda 
(cf. (12)). 

Let us underline that this covariance analysis properly takes into account the 
usually strong geometric nonlinearity in shape analysis via the transfer of geometric 
shape variation to elastic stresses on the average shape, based on paradigms 
from nonlinear elasticity. Displacements or stresses are interpreted as the proper 
linearization of shapes. In abstract terms, either the space of displacements or 
stresses can be considered as the tangent space of shape space at the average shape, 
where the identification of displacements and stresses via (13) provides a suitable 
physical interpretation of stresses as shape variations. 


The impact of the chosen metric. Naturally, the modes of variation depend on the 
chosen inner product. We have already mentioned that in order to be physically 
meaningful, the inner product should act on displacements ux, of the compound 
object (which is composed of all deformed input shapes). If instead the uz, were 
obtained by applying the boundary stresses 0; v to an object which just looks like the 
average shape but does not contain the information how strongly the input shapes 
had to be deformed to arrive at the average, we obtain a different result (Fig. 13, 
left): If the prestressed state of some object regions is neglected, it becomes easier 
to deform them which causes the prediction of stronger variations. Figure 13 also 
hints at the differences between the employed metrics: The L?-metric pronounces 
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Fig. 13 First three dominant modes of variation for six input shapes (left), based on different 


metrics. Left: L-metric on displacements of a non-prestressed object (modes w; with ratios of 
1, 0.23, 0.07). Middle: L?-metric on displacements of the compound object (3 = 1, 0.28, 0.03). 


Right: energy Hessian-based metric on displacements of the compound object (3 = 1,0.61, 0.24) 


shape variations with large displacements even though they are energetically cheap 
(e.g., a rotation of some structure around a joint), while the Hessian of the elastic 
energy measures distances between displacements solely based on the associated 
change of elastic energy. Thus, displacements are weighted strongly in regions and 
directions which are significantly loaded. 


The impact of the nonlinear elasticity model. Likewise, the particular choice of 
the nonlinear elastic energy density has a considerable effect on the average shape 
and its modes of variation. Figure 14 has been obtained using W(D¢) = _ (Dell? + 
4 det Do” — (u + 4) logdet Do — pw — 4, where p and A are the coefficients of 
length and volume change penalization, respectively. A low penalization of volume 
changes apparently leads to independent compression and inflation at the dumbbell 
ends (left), while for deformations with a strong volume change penalization (right), 
material is squeezed from one end to the other. Here, the underlying metric is the 
based on the Hessian of the energy. 

Figures 15-17 show the dominant modes of variation for the examples from the 
previous section. A statistical analysis of the hand shapes in Fig. 15 has also been 
performed in [16] and [28], where the shapes are represented as vectors of landmark 
positions. The average and the modes of variation are quite similar, representing 
different kinds of spreading the fingers. The dominant modes of variation for a set 
of 48 three-dimensional kidney shapes is depicted in Fig. 16, where for all modes 
wz, we show the average (middle) and its variation according to dw; for varying 
6. Local structures seem to be quite well represented and preserved during the 
averaging process and the subsequent covariance analysis compared to, e.g., the 
PCA on kidney shapes in [26] where a medial representation is used. 
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Fig. 14 First three modes of variation for eight dumbbell shapes, /eft for a 100 times stronger 
penalization of length than of volume changes (with ratios a of 1, 0.22, 0.05), right for the reverse 


(# = 1,0.41, 0.07). Each row represents the variation of the average (middle shape) by dw, for 
the mode w;, and varying 6 


dtteeteg TT) 


Fig. 15 First four modes of variation with ratios a of 1, 0.88, 0.42, and 0.25 for the 18 hand 
silhouettes from Fig. 10 
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Fig. 16 Forty-eight input kidneys (Courtesy of Werner Bautz, radiology department at the 
University Hospital Erlangen, Germany) and their first four modes of variation with ratios i of 1, 
0.72, 0.37, and 0.31 


The PCA of the 24 ft shapes from Fig. 12 is shown in Fig. 17 and is much more 
intuitive than the color coding in Fig. 12. The first mode apparently represents 
changing foot lengths, the second and third mode belong to different variants of 
combined width and length variation, and the fourth to sixth mode correspond 
to variations in relative heel position, ankle thickness, and instep height. Finally, 
Fig. 18 shows that the approach also works for image morphologies instead of 
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Fig. 17 The first six dominant modes of variation for the feet from Fig. 12 
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Fig. 18 Eight thorax CT scans from different patients (Courtesy of Bruno Wirth, urology 
department at the Hospital zum hl. Geist, Kempen, Germany) and their first three modes of 
variation with ratios zi of 1, 0.12, and 0.07. Note that the thin lines which can be seen left of 
the heart correspond to contours of the liver, which are only visible in the first and last input image 


shapes, using thorax CT scans as input. Here, the image edge set is considered as the 
corresponding shape, which is typically quite complex and characterized by nested 
contours. The first mode of variation represents a variation in chest size, the next 
mode corresponds to a change of heart and scapula shape, while the third mode 
mostly concerns the rib position. 


Viscous Fluid-Based Shape Space 


As explained in section “Path-Based, Viscous Riemannian Setup,” the viscous fluid 
shape space is by construction a (infinite-dimensional) Riemannian manifold and 
as such is based on the computation of shape paths as opposed to state-based 
approaches like the elastic shape space from the previous section. In the elastic, 
state-based approach, we have to find for each pair of shapes S4 = dO, and 
Sg = 00g one single optimal matching deformation ¢ : O4 — IR¢ via which 
the similarity between S4 and Sz is determined. In contrast, here we require more 
information to measure the distance between the two shapes, namely, an optimal 


1844 M. Rumpf and B. Wirth 


velocity field v(t) : O(t) —> R¢ at each time ¢ within the given time interval [0, 1]. 
In effect, this implies an increase of the dimension of the variational problem by the 
time component. 

The two qualitatively different types of coordinates, the space coordinates (that 
span the space in which the shapes lie) and the time coordinate, are intuitively 
treated in different ways. One possibility is to regard the variational problem of 
computing a geodesic as a classical elliptic boundary value problem in time, in 
which each shape on a path seeks to be in equilibrium with its local neighborhood 
on the path. The equilibrizing force can be interpreted as an acceleration acting on 
the velocity field v. In this setting, it seems most natural to discretize first the time 
variable and approximate geodesics in shape space as discrete sequences So,..., Sx 
of shapes, where each shape is connected to and equilibrates with its neighbors 
and the path length along the discrete path So,...,Sx is approximated as a sum 
aa d (Sx—1, Sx) of approximations d (S;—1, S;,) of the geodesic distance between 
neighboring shapes. The distance d can be based ona matching deformation energy 
which will be elaborated on further down. 

An alternative view starts from the underlying velocity field which generates the 
geodesic. Dupuis et al. [23] and Beg et al. [3] consider shapes (or rather images) 
embedded in a domain 2 C R¢%. These shapes deform according to smooth, 


compactly supported velocity fields v € L? (0. 1]; We? RA )) withn >2+ 4. 
The regularity of the velocity fields is ensured by defining the path dissipation 


as fo Jo Lv-vdx dt and the path length as Vi VJ Lv-vdx dt for a differential 
operator L of sufficiently high order (cf. section “Path-Based, Viscous Riemannian 
Setup”). The corresponding shape deformation ¢ which is induced by the velocity 
field is obtained as the solution ¢ = ¢, of the pointwise, Lagrangian ordinary 
differential equation £6, (x) = v(¢;(x), f). 

In the first approach, the computation of a geodesic was seen as the concatenation 
of a number of local subproblems, each of which represents the approximation of a 
geodesic segment between two intermediate shapes and each of which thus inherits 
the constraint that one shape is transferred exactly into the other. In contrast, in the 
second approach, we have one single constraint, acting at the end of the geodesic 
and expressing that the accumulated flow ¢ deforms the starting shape S,4 into the 
final shape Sg, d(S,4) = Sp. 

Let us now focus on the first approach in which a geodesic path will be approx- 
imated via a finite sequence of shapes So,...,Sx, connected by deformations 
be 2 Opn-1 > R¢ which are optimal in a variational sense and fulfil the constraint 
x (Sk-1) = Sx. 

Given two shapes S4, Sg in some given space of shapes S, we define a 
discrete path of shapes as a sequence of shapes So,...,Sx € S with So = Sy 
and Sx = Sz. For the time step t = % the shape S, is supposed to be 
an approximation of S(q) with %=kt, where (S(t)),¢jo,1) is a continuous path 
connecting S4 = S(O) and Sg = S(1). For each pair of consecutive shapes 
S,—; and Sx, we now consider a matching deformation o, : Ox—1 > R¢ which 
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satisfies dg(Sp—1) = Sz. With each deformation ¢;, we associate a deformation 
energy WactomlPk, Ox—1] = So, W(D¢,;) dx of the same type as described in 
section “State-Based, Path-Independent Elastic Setup.” If appropriately chosen, 
this energy will ensure sufficient regularity and a 1-1 matching property for 
deformations ¢, with finite energy. As in elasticity, the energy is assumed to depend 
only on the local deformation, reflected by the Jacobian D@. Yet, different from 
elasticity, we suppose the material to relax instantaneously so that object O; is again 
in a stress-free configuration when applying ¢,+ at the next time step. Let us also 
emphasize that the stored energy does not depend on the deformation history as 
in most plasticity models in engineering. This energy is now employed to define 
time-discrete counterparts to the dissipation and length of continuous paths from 
section “Path-Based, Viscous Riemannian Setup.” 


Definition 3 (Discrete dissipation and discrete path length). Given a discrete 
path So,...,Sx €S, its dissipation is defined as 


K 
1 
Diss,(So,...,SK) := ) —Weeiaml Pi Ox-1], 
k=1 


where ox : Ox; — R¢ is a minimizer of the deformation energy Wetom|{Px, *] 
under the constraint $; (S,;—1) = S,. Furthermore, the discrete path length is defined 
as 


K 
Lz (So... Sk) = > VWasiaml Pies Ok—11 - 


k=1 


Let us make a brief remark on the proper scaling factors. The deformation energy 
Wetorm(Pks Ox—1] is expected to scale like t? (cf. (7)). Hence, the factor 1 ensures the 
discrete dissipation measure to be conceptually independent of the time step size. 
The same holds for the discrete length measure L,(So,...,Sx). 

To ensure that the above-defined dissipation and length of discrete paths in 
shape space are well defined, a minimizing deformation ¢,; of the elastic energy 
Waeerorml *; Ox—1] with Oe (Sp—1) = Sy has to exist. In fact, this holds for objects 
O;x—1 and O; with Lipschitz boundaries S;_; and S; for which there exists at least 
one bi-Lipschitz deformation dbx of Ox, into O; fork = 1,..., K [83]. 

With the notion of dissipation at hand, we can define a discrete geodesic path 
following the standard paradigms in differential geometry. 


Definition 4 (Discrete geodesic path). A discrete path Sp, S,...,Sx in a set of 
admissible shapes S connecting two shapes S4 = Sp and Sg = Sx is a discrete 
geodesic if there exists an associated family of deformations (¢,),=),...x such 
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Fig. 19 Discrete geodesics between a straight and a rolled up bar, from first row to fourth row 
based on one, two, four, and eight time steps. The light gray shapes in the first, second, and third 
row show a linear interpolation of the deformations connecting the dark gray shapes. The shapes 
from the finest time discretization are overlayed over the others as thin black lines. In the last row, 


the rate of viscous dissipation is rendered on the shape domains ©,,..., © 7 from the previous row, 
color coded as 
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that (¢;, Sk)x=1,...K minimize the total energy Sy 1 Waeeiorm (Bk O;—1] over all 
intermediate shapes S, = 00; cor Sei = dOx—, € Sand all possible matching 
deformations ¢),...,@x with by (S¢_-1) = S_ fork =1,...,K. 


Examples of discrete geodesics are provided in Figs. 19 and 20. Apparently, the 
frame indifference and the (local) injectivity property of the matching deformations, 
which are ensured by the nonlinear deformation energy Wrefom, allow the compu- 
tation of reasonable discrete geodesics with only few intermediate shapes. Under 
sufficient growth conditions on the integrand of the deformation energy Weefom, the 
existence of discrete geodesics is guaranteed at least for certain compact sets S of 
admissible shapes, e.g., shapes S which can be described by spline curves with a 
finite set of control points from some compact domain and which satisfy a uniform 
cone condition in the sense that each x € S is the tip of two cones with fixed height 
and opening angle which lie completely on either side of S [83]. Such requirements 
on S are necessary since the known regularity theory for deformation energies of the 
employed type does not allow to prove Lipschitz regularity of optimal deformations 
so that the intermediate shapes might degenerate. 

The discrete dissipation as the sum of matching deformation energies indeed 
represents an approximation to the time-continuous dissipation of a velocity field 
from section “Path-Based, Viscous Riemannian Setup.” If a smooth path in shape 
space is considered which is interpolated at discrete times % = kt,k = 0,...,K 
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Fig. 20 Discrete geodesic between a cat and a lion and between the hand shapes m336 and m324 
from the Princeton Shape Benchmark [72]. For both examples, the local dissipation is color coded 
on slices through the shapes as 


and if fort € [t,-1, t¢), vr(t) = (uD) o (#14 x) denotes the velocity 
field which generates the associated matching deformations ¢;, then as the time 
step size tT = a decreases and v, converges against a smooth velocity field v, the 
discrete dissipation converges against the time-continuous dissipation (2) induced 
by v (cf. [83] for details). 

Within this framework of geodesics in shape space, the strict constraints that one 
shape is deformed exactly into another one are often inadequate in applications as 
has already been discussed in section “State-Based, Path-Independent Elastic Setup” 
for the state-based, elastic setup. For the computation of an elastic dissimilarity mea- 
sure, the single matching constraint could be relaxed as a mismatch penalty. In the 
Riemannian, viscous setting, we pursue the same concept; however, the particular 
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Fig. 21 Discrete geodesic between the straight and the folded bar from Fig. 19, where the black 
region of the initial shape is constrained to be matched to the black region of the final shape. 
The bottom row shows a color coding of the corresponding viscous dissipation. Due to the strong 


change in relative position of the black region, the intermediate shapes exhibit a strong asymmetry 
and high dissipation near the bar ends 


form of the employed constraints depends on the chosen view on shape geodesics. 
In the framework of geodesics as paths of diffeomorphisms, which we introduced 
at the beginning of this section, there is the single constraint 6(S4) = Sg, meaning 
that the induced diffeomorphism ¢ maps the initial shape S, onto the final shape 
Sz. This constraint can be relaxed in the same manner as in section “State-Based, 
Path-Independent Elastic Setup” via a penalty measuring the mismatch of the shapes 
or of the corresponding objects. For the time-discrete geodesic setting, we have 
a sequence of matching constraints $,(Sp-1) = Sx, kK = 1,..., K, each of 
which can again be relaxed by the same means. In fact, we add to the discrete 
dissipation of a set (%x)x=1,... x Of deformations a sum of mismatch penalties 
yy vol (Op-1 Ady (Ox). In the limit for vanishing time step size tT = z and 
under the same conditions as above, this sum can be shown to converge against 
the optical flow-type functional Ir | (1, v(t)) -2 [t, S(t)] | da for the unit outward 
normal n[t, S(t)] to the space time shape tube 7 = Usefo,1 {t} x S(t). Furthermore, 
ey TL[S,] with L[S,] = H7~!(S;,) has been employed as regularization, which 
in the limit for t > 0 converges against the integral is H4-!(S(t)) dt. 

Real-world objects are most often not only characterized by their outer contour 
but also contain internal structures that have to be matched properly when comput- 
ing the similarity between two objects. As an example, consider the straight and the 
folded rod in Fig. 21. The rods consist of three distinct components, which imposes 
a constraint on reasonable connecting paths: Each component is to be mapped 
onto its correct counterpart. A shortest path under this constraint obviously differs 
significantly from the geodesic which just matches the outer contours (cf. Fig. 19). 

This observation calls for a generalization of shapes, an example of which we 
have already seen in the context of an elastic shape space in Fig. 18, where the edge 
set of an image was considered as a shape. Here, let us adopt a slightly different 
approach and regard shapes as being composed of a number of subcomponents. 
In detail, instead of a geodesic between just two shapes S4 = dO, and Sg = 
dOg, we now seek a geodesic path (S'(t)) _ with S'(t) = dO'(t) fort € 
i=1,...,m with Si (0) = 
with S',(t) = dO',(t). The geodesic path is supposed to 


i=1,..., 7 


[0, 1], between two collections of m separate shapes, (S‘,) 
d0',(t) and (S‘) 


i=1,...,m 
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Fig. 22 Top: frames from a real video sequence of a white blood cell among a number of red ones 
(Courtesy of Robert A. Freitas, Institute for Molecular Manufacturing, California, USA). Middle: 
computed discrete geodesic between the segmented shapes in the first and the last frame. Bottom: 
pushforward of the initial (first four shapes) and pullback of the final frame (last five shapes) 
according to the geodesic flow 


be generated by a joint motion field v(t) : L7L, O'(t) > R¢. The single objects 
O'(t) can then be regarded as the subcomponents of an overall object )7_, O' (t). 
The total dissipation along the path is measured exactly as before by 


. a 
Diss [0.x io = | [ ee (tre[v))?+ tr (e[v2) dx dr. 


This naturally translates to the discrete dissipation of a path with K + | intermediate 


shape collections (Sj),_, _,,.k =0,...,K 
K 
Yan (Miernonl =D fa oy MEADE 
par Ui=1 Op-1 
where the deformations ¢, satisfy the constraints dy (Sia) = Stork = 
1, ,K,i=1, :m,and §, = 8), S-= mil, ,m« 


The different abject components can of eoune be aesiened different material 
properties. Figure 22 shows frames from a real video sequence of moving white 
and red blood cells (top) as well as a discrete geodesic between the first and last 
frame (middle) for which the material parameters of the white blood cell were 
chosen twenty times weaker than for the red blood cells. The result is a nonlinear 
interpolation between distant frames which is in good agreement with the actually 
observed motion. Once geodesic distances between shapes are defined, one can 
statistically analyze ensembles of shapes and cluster them in groups based on the 
geodesic distance as a reliable measure for the similarity of shapes. Two exemplary 
examples are provided by the evaluation of geodesic distances between different 2D 
letters (Fig. 23, left) and between six different 3D foot shapes (Fig. 23, right). In the 
2D example, we clearly identify three distinct clusters (Bs, Xs, and M). 
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Fig. 23 Left: pairwise geodesic distances between (also topologically) different letter shapes. 
Right: pairwise geodesic distances between different scanned 3D feet. The feet have volumes 
499.5, 500.6, 497.6, 434.7, 432, and 381 cm?, respectively 


A Collection of Computational Tools 


So far, we have investigated some of the many aspects on mathematical models 
in shape space without any discussion of the corresponding computational tools 
and numerical algorithms. Hence, let us at least briefly mention some fundamental 
computational aspects to effectively deal with general classes of shapes as boundary 
contours of volumetric objects. 

At first, we replace the strict separation between material inside the object and 
void outside by substituting the void with a material which is several orders of 
magnitude softer than inside the object. This relaxation is important with respect 
to the existence analysis and the stabilization of the computational method. In fact, 
we replace the deformation energy Weim, O] = fe W(D¢) dx by the energy 
Wileonlp, O] = te (a —nXot n) W(D@) dx for a small constant 7. In the 
implementation which underlies the above applications, for 7 = 10~* one observes 
no significant qualitative impact of this regularization on the solution. Furthermore, 
as mentioned above, to ensure regularity of the shape contour S, we take into 
account the area functional L[S] = ¢ da as a prior, weighted with a small factor. 

Compared to a parametric description of shapes, e.g., as a polygonal line or a 
triangulated surface, an implicit description has several advantages. In particular, 
it does not require a remeshing even in case of large deformations, it allows for 
topological transitions without any extra handling of the associated singularities, and 
it can be combined with multi-scale relaxation schemes for an efficient minimization 
of the involved functionals. 

In what follows, we consider a level set and a phase field description of shapes 
and outline the general framework of a multi-scale method based on finite element 
calculus. In fact, the phase field model has been used in the examples for the 
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elastic shape averaging and the PCA, whereas the level set method has served as 
a numerical building block for the computation of time-discrete shape geodesics. 


Shapes Described by Level Set Functions 

The level set method first presented by Osher and Sethian [60] has been used 
for a wide range of applications [59,71]. Burger and Osher gave an overview in 
the context of shape optimization [7]. To numerically solve variational problems 
in shape space, we assume a shape S to be represented by the zero level set 
{x € 2 : u(x) = 0} of a scalar function wu : 82 — R on a computational domain 
2 Cc R%. Furthermore, the zero super level set {x € Q : u(x) > 0} defines the 
corresponding object domain ©. This shape description can be incorporated in 
a variational approach following the approximation proposed by Chan and Vese 
[9]. In fact, the partition of the domain {2 into object and background is encoded 
via a regularized Heaviside function H, o u. As in [9], we consider the function 
A(x) := 5 + + arctan (=) , where ¢ is a scale parameter representing the width 
of the smeared-out shape contour. Then, a deformation energy Wraonm [Ps O| = 


to (1 —nXor n) W(D¢@) dx is approximated by 
Wilaldu] = f (=n) He(u) +) WP) dx. 
2 


Furthermore, the energy F[S4,¢,Sg] = vol(OsA¢7!(Oz )) measuring the 
volumetric mismatch between an object O 4 and the pullback of an object Og under 
a deformation ¢ can be approximated by 


ee [ (H,liew'0'@) = Fila)? ax, 


where u4, ug are level set representations of the shapes S4 and Sz, respectively. 
Finally, the surface area of a shape S, which appears as a prior, is replaced by the 
total variation of H, o u, and we obtain 


cu] = [IV He(lae. 


Let us emphasize that in the actual energy minimization algorithm, the guidance of 
an initial zero level set toward the final shape relies on the nonlocal support of the 
derivative of the regularized Heaviside function (cf. [8]). 


Shapes Described via Phase Fields 

An alternative to a level set description of shapes is a phase field representation. 
Physically, the phase field approach is inspired by the observation that interfaces are 
usually not sharp but characterized by a diffusive transition. Mathematically, there 
are two basic types of such phase field representations, a single phase approach 
as the one presented by Ambrosio and Tortorelli [1] for the approximation of the 
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Mumford-Shah model [56] and the double phase approach by Modica and Mortola 
[55] used to approximate surface integrals. In the shape context studied here, let 
us focus on the single phase model. Thus, a shape S is encoded by a continuous, 
piecewise smooth phase field function u : £2 — R which is zero on S, but close to 
one everywhere else. The specific profile of the phase field function u for a shape S 
is determined via the phase field approximation 


1 1 
L*[u] = 5 felvur + wun 1)° dx 


of the involved surface area sda. As in the above level set model, the phase 
field parameter ¢ determines the width of the diffusive interface. Different from 
the level set model by Chan and Vese, the interface profile is not explicitly 
prescribed but implicitly encoded in the variational approach as the profile attained 
by minimizers of the functional. Based on this phase field model the penalty 
functional F[S4,¢,Sg] = H¢~-'(S,4A@~|(Spg)) measuring the area mismatch 
between a shape Sy, and the pullback of a shape Sg under a deformation @ can 
be approximated by 


F*[ua,, ug] = = [un of)°(1 —u4)? + (1 — ug of) dx, 


where u4, ug are phase fields representing the shapes S4 and Sz, respectively. In 
this type of models, the deformation energy Worm [¢, O] cannot be realized based on 
a phase field function u due to the fact that a single phase model allows to identify 
the shape itself but does not distinguish its inside and outside. Therefore, in the 
presented applications of elastic shape averaging and the elastic PCA, the input 
objects and thus their characteristic functions ¥ were given a priori. 


Multi-scale Finite Element Approximation 

For the spatial discretization of the functionals in the above variational approaches, 
the finite element method can be applied. Hence, the level set function or the phase 
field u, representing a (unknown) shape S, and the different components of the 
deformations ¢ are represented by continuous, piecewise multilinear (trilinear in 3D 
and bilinear in 2D) finite element functions U and @ on a regular grid superimposed 
on the domain 2 = [0, 1]¢. For the ease of implementation, a dyadic grid resolution 
with 24 + 1 vertices in each direction and a grid size h = 2~” is chosen. 


Descent algorithm. The functionals depend nonlinearly both on the discrete 
deformations ® (due to the concatenation U o @ and the nonlinear integrand W(- ) 
of the deformation energy) as well as on the discrete level set or phase field functions 
U (e.g., due to the concatenation of the level set function with the regularized 
Heaviside function H,(-)). In our energy relaxation algorithm for fixed grid size, we 
employ a gradient descent approach. We constantly alternate between performing a 
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single gradient descent step for all deformations and the level set or phase field 
functions. 


Numerical quadrature. Integral evaluations in the descent algorithm are per- 
formed by Gaussian quadrature of third order on each grid cell. For various terms, 
we have to evaluate pullbacks U o @ of a discretized level set function U or a 
test function under a discretized deformation ®. Let us emphasize that quadrature 
based on nodal interpolation of U o ® would lead to artificial displacements near 
the shape edges accompanied by strong artificial tension. Hence, in our algorithm, 
if (x) lies inside 2 for a quadrature point x, then the pullback is evaluated exactly 
at x. Otherwise, we project ®(x) back onto the boundary of {2 and evaluate U at 
that projection point. 


Cascadic multi-scale algorithm. The variational problem considered here is 
highly nonlinear, and for fixed time step size, the proposed scheme is expected to 
have very slow convergence; also it might end up in some nearby local minimum. 
Here, a multilevel approach (initial optimization on a coarse scale and successive 
refinement) turns out to be indispensable in order to accelerate convergence and 
not to be trapped in undesirable local minima. Due to our assumption of a dyadic 
resolution 2“ + 1 in each grid direction, we are able to build a hierarchy of grids with 
2! + 1 nodes in each direction for / = L,...,0. Via a simple restriction operation, 
we project every finite element function to any of these coarse grid spaces. Starting 
the optimization on a coarse grid, the results from coarse scales are successively 
prolongated onto the next grid level for a refinement of the solution [5]. Hence, 
the construction of a grid hierarchy allows to solve coarse scale problems in our 
multi-scale approach on coarse grids. Since the width ¢ of the diffusive shape 
representation should naturally scale with the grid width h, we choose ¢ = h. 


5 Conclusion 


Let us close with a comparison of path- and state-based shape space. Already in 
section “Conceptual Differences Between the Path- and State-Based Dissimilarity 
Measures,” we have studied the difference between the state-based dissimilarity 
measure dj, and the path-based distance d,;,.,;- Based on the applications consid- 
ered in the previous sections, let us compare the underlying concepts now more on 
a conceptual level of the geometry of shape space: 


¢ Non-uniqueness of shape averages. Due to the nonlinearity of the elastic varia- 
tional problem, local minimizers of the elastic energy might be nonunique. There 
might even exist different minimizing deformations with the same elastic energy. 
Mechanically, this nonuniqueness is frequently associated with different buckling 
modes, which occur in case of large, geometrically nonlinear deformations. 
Hence, the shape average need not be uniquely defined, except in the small 
displacement case, where a linear elastic model (8) applies. In case of the path- 
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based approach, geodesics (shortest) do not have to be unique either. Indeed, a 
geodesic is the unique shortest path only until the first conjugate point. Hence, the 
shape average is in a strict sense not well defined if the distances are sufficiently 
large. 

¢ Different physical interpretation of the PCA. In the Riemannian setup with the 
metric being the rate of viscous dissipation, the log; S, corresponds to the initial 
velocity vy, : S — R¢ in the (optimal transport) flow of © associated with 
shape S into O; associated with the kth input shape Sx. In the elastic model, the 
boundary stress 0,0 : OO > R¢ results from the deformation dy of Ox onto 
the average object O and effectively is the restoring force acting on the average 
shape S. Via the linearized elasticity problem in the prestressed compound 
configuration of the average object O, these restoring forces are identified with 
displacements uz. Depending on the model, either the flow velocities vx, or 
the linear elastic displacements u,; form the basis of a covariance analysis in 
the linear vector space of mappings O — IR. The outcome are principal 
shape variations of the average shape, either generated by motion fields or 
displacements, respectively. 

* Quantitative shape analysis. The Riemannian metric given by the rate of viscous 
dissipation in the path-based viscous fluid approach allows direct comparison of 
multiple ensembles of shapes via pairwise distance computations. Due to the lack 
of a triangle inequality, this is possible only in a restricted sense in the state-based 
elastic approach, where dissimilarity measures for one fixed shape and a set of 
varying shapes can be computed. 

¢ The method of choice depends on the specific application. If shapes are 
considered as boundaries of objects with a viscous fluid inside, then the path- 
based approach would be more appropriate. The state-based elastic approach is 
favorable for objects which behave more like deformable solids. 
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Abstract 


Nonrigid shapes are ubiquitous in nature and are encountered at all levels of life, 
from macro to nano. The need to model such shapes and understand their 
behavior arises in many applications in imaging sciences, pattern recognition, 
computer vision, and computer graphics. Of particular importance is understand- 
ing which properties of the shape are attributed to deformations and which are 
invariant, i.e., remain unchanged. This chapter presents an approach to nonrigid 
shapes from the point of view of metric geometry. Modeling shapes as metric 
spaces, one can pose the problem of shape similarity as the similarity of metric 
spaces and harness tools from theoretical metric geometry for the computation 
of such a similarity. 


1 Introduction 


Those who played the game Rock, Paper, and Scissors in their childhood certainly 
remember the three gestures used in the game: Rock, represented by a clenched fist; 
Paper, represented by an open hand; and Scissors, represented by the extended index 
and middle fingers. These gestures are a toy example of the nonrigid shape similar- 
ity problem, which is the central topic of this chapter. No matter how one bends the 
fingers, he will immediately recognize the underlying object: the human hand. 

More generally, the problem of determining the similarity of shapes undergoing 
certain class of transformations is termed invariant shape similarity. A similarity 
criterion is said to be invariant if it is not influenced by the transformation (Fig. 1). 
Different classes of transformations prescribe different similarity criteria based 
on geometric shape properties that are invariant under such transformations. The 
wider is the class, the less properties are preserved, and as a thumb rule, the 
more difficult is the problem. Specifically, in this chapter, we will consider rigid, 
inelastic, topology-changing, and scaling transformations. In many cases, such 
transformations are a good approximation of real transformations that natural 
objects may undergo. 
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Fig. 1 Invariant shape 
similarity 


SIMILARITY 


Problems 


Since nonrigid shapes are ubiquitous in the world and are encountered at all scales 
from macro to nano, nonrigid shape similarity plays a key role in many applications 
in imaging sciences, pattern recognition, computer vision, and computer graphics. 
Two archetype problems in shape analysis considered in this chapter are invariant 
similarity and correspondence. As will be discussed in the following, these two 
problems are interrelated: finding the best correspondence between two shapes also 
allows quantifying their similarity. 

A good example of shape similarity is the problem of face recognition [19,21,24]. 
As the crudest approximation, one can think of faces as rigid surfaces and compare 
them using similarity criteria invariant under rigid transformations. However, such 
an approach does not account for surface deformations due to facial expressions, 
which can be approximated by inelastic deformations. Accounting for such defor- 
mations requires different similarity criteria. Yet, even elastic deformations are 
not enough to model the behavior of human faces: many facial expresses involve 
elastic deformations that change the facial shape topology (think of open and closed 
mouth). This extension of the model will require revisiting the similarity criterion 
once again. 

The problem of correspondence is often encountered in shape synthesis appli- 
cations such as morphing. In order to morph one shape into the other, one 
needs to know which point on the first shape will be transformed into a point 
on the second shape, in other words, establishing a correspondence between the 
shapes. 
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Methods 


Many different approaches to shape similarity and correspondence can be consid- 
ered as instances of the minimum-distortion correspondence problem, in which two 
shapes are endowed with certain structure, and one attempts to find the best (least 
distorting) matching between these structures. Such structures can be local (e.g., 
multiscale heat kernel signatures [106], local photometric properties [107, 119], or 
conformal factor [12]) or global (e.g., geodesic [24, 47, 78], diffusion [28], and 
commute time [31]) distances. The distortion of the best possible correspondence 
can be used as a criterion of shape similarity. By defining a structure invariant under 
certain class of transformations, it is possible to obtain invariant correspondence or 
similarity. 

Local structures can be regarded as feature descriptors. As a model for global 
structures, metric spaces are used. 


Chapter Outline 


This chapter tries to present a unifying view on the archetypical problems in 
shape analysis. The first part presents a metric view on the problem of nonrigid 
shape similarity and correspondence, a common denominator allowing to deal with 
different types of invariance. According to this model, shapes are represented as 
metric spaces. The mathematical foundations of this model are provided in Sect. 2. 
Sections 3 and 4 deal with discrete representation of shapes, which is of essence 
in practical numerical computations. Section 5 provides a rigorous formulation 
of the invariant shape similarity problem and reviews different algorithms for its 
computation. Section 6 deals with an extension of invariant similarity to shapes 
which are partially similar, and Sect. 7 deals with a particular case of self-similarity 
and symmetry. Local feature-based methods and their use to create global shape 
descriptors are presented in Sect. 8. Finally, concluding remarks in Sect. 9 end the 
chapter. This chapter is based in part on the book [26], to which the reader is referred 
for further discussion and details. 


2 Shapes as Metric Spaces 


Elad and Kimmel [47, 48], Mémoli and Sapiro [78], and Bronstein et al. [23, 24] 
suggested to model shapes as metric spaces. The key idea of this model is that it 
allows to compare shapes as metric spaces. Since the model allows arbitrarity in the 
definition of the metric, desired invariance considerations guide the choice of the 
metric. 

This section introduces the mathematical formalism and notation of this model 
and shows the construction of three different types of metric geometries: Euclidean, 
Riemannian, and diffusion. 
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Basic Notions 


Topological Spaces 
Given a set X, a topology T on X is a collection of subsets of X satisfying (Ti) 
X,0 € T; (Tii) U, Ua € T for Uy € T; and (Tiii) (sy U; ¢€ T forU; ¢ T. X 
together with T is called a topological space. By convention, sets in T are referred 
to as open sets and their complements as closed sets. 

A neighborhood N(x) of x is a set containing an open set U € T such that 
x € U. Points with neighborhood are called interior. 

A topological space is called Hausdorff if distinct points in it have disjoint 
neighborhoods. 

Two topological spaces X and Y are homeomorphic if there exists bijection 
a : X — Y which is continuous and has continuous inverse a~!. Since homeo- 
morphisms copy topologies, homeomorphic spaces are topologically equivalent [1]. 


Metric Spaces 
A function d : X x X — R which is (Mi) positive-definite (d(x, y) > 0 for all 
x # y and d(x, y) = 0 for x = y) and (Mii) subadditive (d(x,z) < d(x,y) + 
d(y,z) for all x, y, z) is called a metric on X. The metric is an abstraction of the 
notion of distance between pairs of points on X. Property (Mii) is called triangle 
inequality and generalizes the known fact: the sum of the lengths of two edges of a 
triangle is greater or equal to the length of the third edge. The pair (X, @) is called 
a metric space. 

A metric induces topology through the definition of open metric ball B,(x) = 
{x’ € X : d(x, x’) <r}. A neighborhood of x in a metric space is a set containing 
a metric ball B, (x) [34]. 


Isometries 
Given two metric spaces (X, d) and (Y, 5), the set C C X x Y of pairs such that for 
every x € X there exists at least one y € Y such that (x, y) € C, and similarly, for 
every y € Y, there exists an x € X such that (x, y) € C is called a correspondence 
between X and Y. Note that a correspondence C is not necessarily a function. The 
correspondence is called bijective if every point in X has a unique corresponding 
point in Y and vice versa. 

The discrepancy of the metrics d and 6 between the corresponding points is 
called the distortion of the correspondence, 


dis(C) = sup |d(x, x’) — 8(y, y’)]. 
(x,y), (x,y EC 


Metric spaces (X,d) and (Y,64) are said to be €-isometric if there exists a 
correspondence C with dis(C) < €. Sucha C is called an €-isometry. 

A particular case of a 0-isometry is called an isometry. In this case, the 
correspondence is a bijection and X and Y are called isometric. 
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Euclidean Geometry 


Euclidean space R” (hereinafter also denoted as E) with the Euclidean metric 
dg(x, x’) = ||x — x’||2 is the simplest example of a metric space. Given as a subset 
X of E, we can measure the distances between points x and x’ on X using the 
restricted Euclidean metric, 


delyxx (x, x’) = dg(x, x’) 


for all x, x’ in X. 

The restricted Euclidean metric dg| xxx is invariant under Euclidean transforma- 
tions of X, which include translation, rotation, and reflection in E. In other words, 
X and its Euclidean transformation i (X ) are isometric in the sense of the Euclidean 
metric. Euclidean isometries are called congruences, and two subsets of E differing 
up to a Euclidean isometry are said to be congruent. 


Riemannian Geometry 


Manifolds 

A Hausdorff space X which is locally homeomorphic to R” (i.e., for every x in 
X, there exists a neighborhood U and a homeomorphism a : U — R’") is called 
an n-manifold or an n-dimensional manifold. The function @ is called a chart. A 
collection of neighborhoods that cover X together with their charts is called an 
atlas on X . Given two charts a and 6 with overlapping domains U and V, the map 
Ba! :a(U NV) > B(U NV) is called a transition function. A manifold whose 
transition functions are all differentiable is called a differentiable manifold. More 
generally a C' -manifold has all transition maps k-times continuously differentiable. 
A C™-manifold is called smooth. 

A manifold with boundary is not a manifold in the strict sense of the above 
definition. Its interior points are locally homeomorphic to R”, and every point on 
the boundary 0X is homeomorphic to [0, 00) x R"!. 

Of particular interest for the discussion in this chapter are two-dimensional 
(n = 2) manifolds, which model boundaries of physical objects in the world 
surrounding us. Such manifolds are also called surfaces. In the following, when 
referring to shapes and objects, the terms manifold, surface, and shape will be used 
synonymously. 


Differential Structures 

Locally, a manifold can be represented as a linear space, in the following way. Let 
a:U — R" beachart on a neighborhood of x and y : (—1, 1) :—> X bea differen- 
tiable curve passing through x = y(0). The derivative of the curve (a o y)(0) is 
called a tangent vector at x. The set of all equivalence classes of tangent vectors at 
x forming an n-dimensional real vector space is called the tangent space T, X at x. 
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A family of inner products (-, -), : TX x T,X — R depending smoothly on x 
is called Riemannian metric tensor. A manifold X with a Riemannian metric tensor 
is called a Riemannian manifold. 

The Riemannian metric allows to define local length structures and differential 
calculus on the manifold. Given a differentiable scalar-valued function f : X > R, 
the exterior derivative (differential) is a form df = (Vf, -}) on the tangent space 
TX. For a tangent vector v € 7,X, df(x)v = (Vf(x),Vv)x. Vf is called the 
gradient of f at x and is a natural generalization of the notion of the gradient in 
vector spaces to manifolds. Similarly to the definition of Laplacian satisfying 


/ (V AVA) sdu(x) = / hAx faplx) 
xX xX 


for differentiable scalar-valued functions f and h, the operator Ay is called the 

Laplace-Beltrami operator, a generalization of the Laplacian. Here, 44 denotes 
the measure associated with the n-dimensional volume element (area element for 
n = 2). The Laplace—Beltrami operator is (Li) symmetric (fy hAy fdu(x) = 
Jy fAxxdu(x)), (Lii) of local action (Ay f(x) is independent of the value of 
f(x’) for x’ ¥ x), and (Liii) positive semi-definite (fy fAx fdu(x) = 0) (in many 
references, the Laplace—Beltrami is defined as a negative semi-definite operator) and 
(Liv) coincides with the Laplacian on Euclidean domains, such that Ay f = Oif f 
is a linear function and X is Euclidean. 


Geodesics 

Another important use of the Riemannian metric tensor is to measure the length of 
paths on the manifold. Given a continuously differentiable curve y : [a,b] > X, its 
length is given by 


p 1172: 
£y) = i Y'O. YO 2 at. 


For the set of all continuously differentiable curves I(x, x’) between the points 
/ 
Hs 


dy(x,x’)= inf €(y) (1) 
yer (x,x’) 


defines a metric on X referred to as length or geodesic metric. If the manifold is 
compact, for any pair of points x and x’, there exists a curve y € I'(x, x’) called a 
minimum geodesic such that £(y) = dy (x, x’). 


Embedded Manifolds 

A particular realization of a Riemannian manifold called embedded manifold (or 
embedded surface for n = 2) is a smooth submanifold of R” (m > n). In this case, 
the tangent space is an n-dimensional hyperplane in R”, and the Riemannian metric 
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is defined as the restriction of the Euclidean inner product to the tangent space, 
(-, -)pm|ry. 

The length of a curve y : [a,b] ~ X Cc R” on an embedded manifold is 
expressed through the Euclidean metric, 


b b 
ey) = / (('O.y' Olnelypx)? at.= / ly Olledt 2) 


and the geodesic metric dy defined according to (1) is said to be induced by dpm. 
(Repeating the process, one obtains that the metric induced by dy is equal to dy. 
For this reason, dy is referred to as intrinsic metric [34].) 

Though apparently embedded manifolds are a particular case of a more general 
notion of Riemannian manifolds, it appears that any Riemannian manifold can be 
realized as an embedded manifold. This is a consequence of the Nash embedding 
theorem [84], showing that a C’(k > 3) Riemannian manifold can be isometrically 
embedded in a Euclidean space of dimension m = n? + 5n + 3. In other words, any 
smooth Riemannian manifold can be defined as a metric space which is isometric to 
a smooth submanifold of a Euclidean space with the induced metric. 


Rigidity 

Riemannian manifolds do not have a unique realization as embedded manifolds. 
One obvious degree of freedom is the set of all Euclidean isometries: two congruent 
embedded manifolds are isometric and thus are realizations of the same Riemannian 
manifold. However, a Riemannian manifold may have two realizations which are 
isometric but incongruent. Such manifolds are called nonrigid. If, on the other hand, 
a manifold’s only isometries are congruences, it is called rigid. 


Diffusion Geometry 


Another type of metric geometry arises from the analysis of heat propagation 
on manifolds. This geometry is called diffusion and is also intrinsic. We start by 
reviewing properties of diffusion operators. 


Diffusion Operators 

A function k : X x X — R is called a diffusion kernel if it satisfies the following 
properties: (Ki) nonnegativity: k(x,x) > 0; (Kii) symmetry: k(x,y) = k(y,x); 
(Kiii) positive-semidefiniteness: for every bounded /, 


/ i Ke, DOF duCOdu(y) = 0: 


(Kiv) square integrability: [ [Pe 2duerduo < oo; and (Kv) conserva- 


tion: | k(x, y)du (y) = 1. The value of k(x, y) can be interpreted as a transition 


probability from x to y by one step of a random walk on X. 
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Diffusion kernel defines a linear operator 


Kee i ko, YFO)an(y); 3) 


which is known to be self-adjoint. Because of (Kiv), K has a finite Hilbert norm 
and therefore is compact. As the result, it admits a discrete eigendecomposition 
Ky; = a; ~; with some eigenfunctions {y;}P%, and eigenvalues {a;}?2,. a; = 0 
by virtue of property (Kiii), and a; < 1 by virtue of (Kv) and consequence of the 
Perron—Frobenis theorem. 

By the spectral theorem, the diffusion kernel can be presented as k(x, y) = 


Co 
> aj Wi (x) Wi (v). Since {y;}2, form an orthonormal basis of L?(X), 


i=1 
i=0 


J [Pe nducoducry = Ya}, (4) 
i=0 


a fact sometimes referred to as Parseval’s theorem. Using these results, properties 
Co 
(Kiii-Kv) can be rewritten in the spectral form as 0 < a; < 1 and a? <O. 


An important property of diffusion operators is the fact that for way t = 0, the 
operator K’ is also a diffusion operator with the eigenbasis of K and corresponding 
eigenvalues i Saas The kernel of K’ expresses the transition probability by 
random walk of t steps. This allows to define a scale space of kernels, {k;(x, y)}rer, 
with the scale parameter f. 

There exist a large variety of possibilities to define a diffusion kernel and the 
related diffusion operator. Here, we restrict our attention to operators describing 
heat diffusion. Heat diffusion on surfaces is governed by the heat equation 


(ax + =) u(x,t) =0; u(x, 0) = uo(x), (3) 


where u(x,t) is the distribution of heat on the surface at point x in time f, uo is 
the initial heat distribution, and Ay is the positive-semidefinite Laplace—Beltrami 
operator, a generalization of the second-order Laplacian differential operator A 
to non-Euclidean domains. (If X has a boundary, boundary conditions should be 
added.) 

On Euclidean domains (X = R”), the classical approach to the solution of the 
heat equation is by representing the solution as a product of temporal and spatial 
components. The spatial component is expressed in the Fourier domain, based on 
the observation that the Fourier basis is the eigenbasis of the Laplacian A, and the 
corresponding eigenvalues are the frequencies of the Fourier harmonics. A particular 
solution for a point initial heat distribution u(x) = 6(x — y) is called the heat 
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case. A general solution for any initial condition uo is given by convolution H’ up = 
Jam A(x — y)uo(y)dy, where H’ is referred to as heat operator. 

In the non-Euclidean case, the eigenfunctions of the Laplace—Beltrami operator 
Ayd¢; = 4;¢; can be regarded as a “Fourier basis” and the eigenvalues given the 
“frequency” interpretation. The heat kernel is not shift invariant but can be expressed 
ash, (x,y) = Dey e gi (x) Gi (9). 

It can be shown that the heat operator is related to the Laplace—Beltrami operator 
as H’ = e ‘4, and as a result, it has the same eigenfunctions ¢; and corresponding 
eigenvalues e~'*', It can be thus seen as a particular instance of a more general 
family of diffusion operators K diagonalized by the eigenbasis of the Laplace— 
Beltrami operator, namely, K’s as defined in the previous section but restricted 
to have the eigenfunctions yw; = ¢;. The corresponding diffusion kernels can be 
expressed as 


kernel h(x — y) = e—le-yIP/4t, which is shift invariant in the Euclidean 


k(x. y) = > KA) oi (2) G0), (6) 


i=0 


where K(A) is some function such that wa; = K(A;) (in the case of H;, K(A) = 
e~'). Since the Laplace—Beltrami eigenvalues can be interpreted as frequency, 
K(A) can be thought of as the transfer function of a low-pass filter. Using this 
signal processing analogy, the kernel k(x, y) can be interpreted as the point spread 
function at a point y, and the action of the diffusion operator K f on a function f 
on X can be thought of as the application of the point spread function by means of 
a non shift-invariant version of convolution. The transfer function of the diffusion 
operator K’ is K‘(A), which can be interpreted as multiple applications of the filter 
K(A). Such multiple applications decrease the effective bandwidth of the filter and, 
consequently, increase its effective support in space. Because of this duality, both 
k(x, y) and K(A) are often referred to as diffusion kernels. 


Diffusion Distances 


Since a diffusion kernel k(x, y) measures the degree of proximity between x and y, 
it can be used to define a metric 


d?(x,y) = [k(x -) —k(, lfacx (7) 


on X, which was first constructed by Berard et al. in [*] and dubbed as the diffusion 
distance by Coifman and Lafon [40]. Another way to interpret the latter distance 
is by considering the embedding VW : x +» L?(X) by which each point x on 
X is mapped to the function V(x) = k(x, -). The embedding W is an isometry 
between X equipped with diffusion distance and L?(X) equipped with the standard 
L? metric, since d(x, y) = ||W(x) — W(y)|lz2(x)- Because of spectral duality, the 
diffusion distance can also be written as 
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d? (x,y) = D> Ki) (Gi (x) — bi(y)’. (8) 


i=0 


Here as well we can define an isometric embedding ® : x + £? with ®(x) = 
{K (Ai) Gi (X)}%20, termed as the diffusion map by Lafon. The diffusion distance can 
be cast as d(x, y) = ||®(x) — ®(y)|le. 

The same way a diffusion operator K’ defines a scale space, a family of diffusion 
metrics can be defined for t > O as 


d? (x,y) = |®:(x) — &O)|le (9) 


= > K* (A; )(bi (x) — gi)’. 


i=0 


where ®,(x) = {K‘(A;)$;(x)}%o. Interpreting diffusion processes as random 
walks, d, can be related to the “connectivity” of points x and y by walks of length 
t (the more such walks exist, the smaller is the distance). 

The described framework is very generic, leading to numerous potentially useful 
diffusion geometries parametrized by the selection of the transfer function K(A). 
Two particular choices are frequent in shape analysis, the first one being the heat 
kernel, K;(A) = e~, and the second one being the commute time kernel, K(A) = 
ree resulting in the heat diffusion and commute time metrics, respectively. While 
the former kernel involves a scale parameter, typically tuned by hand, the latter one 
is scale invariant, meaning that neither the kernel nor the diffusion metric it induces 
changes under uniform scaling of the embedding coordinates of the shape. 


3 Shape Discretization 


In order to allow storage and processing of a shape by a digital computer, it has to 
be discretized. This section reviews different notions in the discrete representation 
of shapes. 


Sampling 


Sampling is the reduction of the continuous surface X representing a shape into a 
finite discrete set of representative points X= {x1,...,Xy}. The number of points 
|X | = N is called the size of the sampling. The radius of the sampling refers to the 
smallest positive scalar r for which X is an r-covering of X, i.e., 


r(X) = max min dx (x, X;). (10) 


xe xjEX 
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The sampling is called s-separated if dy (x;,xj;) = s for every distinct x;,x; € Le 
Sampling partitions the continuous surface into a set of disjoint regions, 


V; = {x €X: dy(x, x) < dy(x, xj), xj4; € X}, (11) 


called the Voronoi regions [7] (Fig.2). A Voronoi region V; contains all the points 
on X that are closer to x; than to any other x;. That the sampling is said to induce a 
Voronoi tessellation (Unlike in the Euclidean case where every sampling induces a 
valid tessellation (cell complex), samplings of curved surfaces may result in Voronoi 
regions that are not valid cells, i.e., are not homeomorphic to a disk. In [66], Leibon 
and Letscher showed that an r-separated sampling of radius r with 7 smaller than 
i of the convexity radius of the shape is guaranteed to induce a valid tessellation.) 
which we denote by V(X) = {V;,..., Vn}. 

Sampling can be regarded as a quantization process in which a point x on the 
continuous surface is represented by the closest x; in the sampling [51]. Such 
a process can be expressed as a function mapping each V; to the corresponding 
(Points on the boundary of the Voronoi regions are equidistant from at least two 
sample points and therefore can be mapped arbitrarily to any of them.) sample x;. 
Intuitively, the smaller are the Voronoi regions, the better is the sampling. Sampling 
quality is quantified using an error function. For example, 


€oo(X) = max dy(x, X) = max min dy(x, x;) (12) 
xeX xEX xj€ x 
determines the maximum size of the Voronoi regions. If the shape is further 


equipped with a measure (e.g., the standard area measure), other error functions 
can be defined, e.g., 


eX) =f dhe xrdnca. (13) 


In what follows, we will show sampling procedures optimal or nearly optimal in 
terms of these criteria. 


Farthest Point Sampling 
Farthest point sampling (FPS) is a greedy procedure constructing a sequence of 


samplings X,, X2,.... A sampling X41 is constructed from Xw by adding the 
farthest point 
X41 = argmaxdy(x,Xy) = argmax min dy(x, xj). (14) 
xEX xeX x;EXy 


The sequence {ry} of the sampling radii associated with {X n} is nonincreasing, 
and, furthermore, each Xy is also ry-separated. The starting point x; is usually 
picked up at random, and the stopping condition can be either the sampling size or 
radius. 
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Fig. 2 Voronoi 
decomposition of a surface 
with a non-Euclidean metric 


Though FPS does not strictly minimize any of the error criteria defined in the 
previous section, in terms of €o9, it is no more than twice inferior to the optimal 
sampling of the same size [59]. In other words, for X produced using FPS, 


€oo(X) <2 min €o0(X’). (15) 
X= 12 


This result is remarkable, as finding the optimal sampling is known to be an NP-hard 
problem. 


Centroidal Voronoi Sampling 
In order for a sampling to be €2-optimal, each sample x; has to minimize 


i dy (x, x;)du(x). (16) 
Vj; 


A point minimizing the latter quantity is referred to as the centroid of V;. Therefore, 
an €2-optimal sampling induces a so-called centroidal Voronoi tessellation (CVT), in 
which the centroid of each Voronoi region coincides with the sample point inducing 
it [45, 90]. Such a tessellation and the corresponding centroidal Voronoi sampling 
are generally not unique. 

A numerical procedure for the computation of a CVT of a shape is known as the 
Lloyd—Max algorithm [68,73]. Given some initial sampling X! of size N (produced, 
e.g., using FPS), the Voronoi tessellation induced by it is computed. The centroids 
of each Voronoi region are computed, yielding a new sampling X? of size N. 
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The process is repeated iteratively until the change of X* becomes insignificant. 
While producing high-quality samplings in practice, the Lloyd—Max procedure is 
guaranteed to converge only to a local minimum of €2. For computational aspects of 
CVTs on meshes, the reader is referred to [90]. 


Shape Representation 


Once the shape is sampled, it has to be represented in a way allowing computation 
of discrete geometric quantities associate with it. 


Simplicial Complexes 

The simplest representation of a shape is obtained by considering the points of the 
sampling as points in the ambient Euclidean space. Such a representation is usually 
referred to as a point cloud. Points in the cloud are called vertices and denoted 
by X = {x,,...,xXy}. The notion of a point cloud can be generalized using the 
formalism of simplical complexes. For our purpose, an abstract k-simplex is a set 
of cardinality k + 1. A subset of a simplex is called a face. A set K of simplices 
is said to be an abstract simplical complex if any face of o € K is also in K, and 
the intersection of any two simplices 01,02 € K is a face of both o; and o2. A 
simplicial k-complex is a simplicial complex in which the largest dimension of any 
simplex is k. A simplicial k-complex is said to be homogeneous if every simplex of 
dimension less than k is the face of some k-simplex. A topological realization K of 
a simplicial complex K maps K to a simplicial complex in R”, in which vertices are 
identified with the canonical basis of R”, and each simplex in K is represented as 
the convex hull of the corresponding points {e;}. A geometric realization ¢x(K) is 
a map of the simplicial complex K to R? defined by associating the standard basis 
vectors e; € IR” with the vertex positions x;. 

In this terminology, a point cloud is a simplicial 0O-complex having a discrete 
topology. Introducing the notion of neighborhood, we can define a subset FE C XxX 
of pairs of vertices that are adjacent. Pairs of adjacent vertices are called edges, and 
the simplicial 1-complex XU E has a graph topology, i.e., the set of vertices X forms 
an undirected graph with the set of edges FE. A simplicial 2-complex consisting of 
vertices, edges, and triangular faces built upon triples of vertices and edges is called 
a triangular mesh. The mesh is called topologically valid if it is homeomorphic to 
the underlying continuous surface X. This usually implies that the mesh has to be 
a two manifold. A mesh is called geometrically valid if it does not contain self- 
intersecting triangles, which happens if and only if the geometric realization ¢x(K) 
of the mesh is bijective. Consequently, any point x on a geometrically valid mesh 
can be uniquely represented as x = gx(u). The vector u is called the barycentric 
coordinates of x and has at most three nonzero elements. If the point coincides 
with a vertex, u is a canonical basis vector; if the point lies on an edge, u has two 
nonzero elements; otherwise, u has three nonzero elements and x lies on a triangular 
face. 
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A particular way of constructing a triangular mesh stems from the Voronoi 
tessellation induced by the sampling. We define the simplicial 3-complex as 


XU {(x%j, xj): Vi NOV; AOU LG, xj, xe) 1 Vi NOV; N OVE A GO}, (17) 


in which a pair of vertices spans an edge and a triple of vertices spans a face if the 
corresponding Voronoi regions are adjacent. A mesh defined in this way is called a 
Delaunay mesh. (Unlike in the Euclidean case where every sampling induces a valid 
Delaunay triangulation, an invalid Voronoi tessellation results in a topologically 
invalid Delaunay mesh. In [66], Leibon and Letscher showed that under the same 
conditions sufficient for the existence of a valid Voronoi tessellation, the Delaunay 
mesh is also topologically valid.) 


Parametric Surfaces 

Shapes homeomorphic to a disk can be parametrized using a single global chart, 
e.g., on the unit square, x : [0, 1]? > R?. (Manifolds with more complex topology 
can still be parametrized in this way by introducing cuts that open the shape into a 
topological disk.) Such surfaces are called parametric and can be sampled directly in 
the parametrization domain. For example, if the parametrization domain is sampled 
on a regular Cartesian grid, the shape can be represented as three N x N arrays of 
x, y, and z values. Such a completely regular structure is called a geometry image 
[56, 69, 99] and can be thought indeed as a three-channel image that can undergo 
standard image processing such as compression. Geometry images are ideally 
suitable for processing by vector and parallel hardware. 


Implicit Surfaces 

Another way of representing a shape is by considering the isosurfaces {x : ®(x) = 
0} of some function ® defined on a region of R*. Such a representation is called 
implicit, and it often arises in medical imaging applications, where shapes are 
two-dimensional boundaries created by discontinuities in volumetric data. Implicit 
representation can be naturally processed using level-set-based algorithms, and it 
easily handles arbitrary topology. A disadvantage is the bigger amount of storage 
commonly required for such representations. 


4 Metric Discretization 


Next step in the discrete representation of shapes is the discretization of the metric. 


Shortest Paths on Graphs 


The most straightforward approach to metric discretization arises from considering 
the shape as a graph in which neighbor vertices are connected. A path in the graph 
between vertices x;,.x; is an ordered set of connected edges 
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TO, x;) = {(Xi,, Xin), (Xi, Xiz)s sates (Xi, Xigey) E, (18) 
where x;, = x; and x;,,, = x;. The length of path I’ is the sum of its constituent 
edge lengths, 

k 
LO (a,x) = Om, — Hal. (19) 


n=1 


A minimum geodesic in a graph is the shortest path between the vertices, 


"(es = arg, doin OE ay): (20) 
Xj Xj 


We can use dz (x;,x;) = L(I'*(x;, x;)) as a discrete approximation to the geodesic 
metric dy (x;,x;). 

According to the Bellman optimality principle [10], given '* (x;, x;) a shortest 
path between x; and x; and x, a point on the path, the sub-paths ['*(x;, x;,) and 
I’* (xx, x;) are the shortest paths between x;, x, and x;, x;, respectively. The length 
of the shortest path in the graph can be thus expressed by the following recursive 
equation: 


di(x;,xj)= min {di (xi, xx) + Ilxx — x; ll}. (21) 
Xk (XK xX; )EE 

Dijkstra’s Algorithm 

A famous algorithm for the solution of the recursion (21) was proposed by Dijkstra. 

Dijkstra’s algorithm measures the distance map d(x) = dzi(x;, Xx) from the source 

vertex x; to all the vertices in the graph. 


Initialize d(x;) = 0, d(xx) = oo for all k ¥ i; queue of unprocessed vertices 
QO = {m,..., xn}. 
while Q is non-empty do 

Find vertex with smallest value of d in the queue 


x = argmind(x) 
xE€Q 
for all unprocessed adjacent vertices x’ € Q : (x, x’) € E do 


d(x’) = min {d(x’), d(x) + ||x — x’ ||} 


end for 
Remove x from Q. 
end while 


Manifold Intrinsic Similarity 1875 


Every vertex in Dijkstra’s algorithm is processed exactly once; hence, Nn outer 
iterations are performed. Extraction of vertex with smallest d is straightforward with 
O(N) complexity and can be reduced to O(log NV) using efficient data structures 
such as Fibonacci heap. In the inner loop, updating adjacent vertices in our case, 
since the graph is sparsely connected, is O(1). The resulting overall complexity is 
O(N log N). 


Metrication Errors and Sampling Theorem 

Unfortunately, the graph distance dz, is an inconsistent approximation of dy, in the 
sense that d;, usually does not converge to dy when the sampling becomes infinitely 
dense. This phenomenon is called metrication error, and the reason is that the graph 
induces a metric inconsistent with dy (Fig.3). While metrication errors make in 
general the use of d; an approximation of dy disadvantageous, the Bernstein—de 
Silva—Langford—Tenenbaum theorem [14] states that under certain conditions, the 
graph metric d; can be made as close as desired to the geodesic metric dy. The 
theorem is formulated as a bound of the form 


d 
hg Se he, (22) 
dy 


where A, A2 depend on shape properties, sampling quality, and graph connectivity. 
In order for d;, to represent dy accurately, the sampling must be sufficiently dense, 
length of edges in the graph bounded, and sufficiently close vertices must be 
connected, usually in a non-regular manner. 


Fast Marching 


Eikonal Equation 

An alternative to computation of a discrete metric on a discretized surface is the 
discretization of the metric itself. The distance map d(x) = dy(xo, x) (Fig. 4) 
on the manifold can be associated with the time of arrival of a propagating front 
traveling with unit speed (illustratively, imagine a fire starting at point xo at time 
t = 0 and propagating from the source). Such a propagation obeys the Fermat 
principle of least action (the propagating front chooses the quickest path to travel, 
which coincides with the definition of the geodesic distance) and is governed by the 
eikonal equation 


|Vxd lo = 1, (23) 


where Vy is the intrinsic gradient on the surface X. Eikonal equation is a 
hyperbolic PDE with boundary conditions d(xo) = 0; minimum geodesics are its 
characteristics. Propagation direction is the direction of the steepest increase of d 
and is perpendicular to geodesics. 
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Fig. 3 Shortest paths measured by Dijkstra’s algorithm (solid bold lines) do not converge to 
the true shortest path (dashed diagonal), no matter how much the grid is refined (Reproduced 
from [25]) 


Since the distance map is not everywhere differentiable (in particular, at the 
source point), no solution to the eikonal equation exists in the classical sense, 
while there exist many non C! functions satisfying the equation and the boundary 
conditions. Among such functions, the largest d satisfying the boundary conditions 
and the inequality 


|Vxdll2 <1 (24) 


at every point where Vxd exists is called the viscosity solution [42]. The viscosity 
solution of the eikonal equation always exists and is unique, and its value at a point 
x coincides with dy (x, Xo). It is known to be monotonous, i.e., not having local 
maxima. 


Triangular Meshes 

A family of algorithms for finding the viscosity solution of the discretized eikonal 
equation by simulated wavefront propagation is called fast marching methods 
[63, 100, 112]. Fast marching algorithms can be thought of as continuous variants of 
Dijkstra’s algorithm, with the notable difference that they consistently approximate 
the geodesic metric dy on the surface. 
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Fig. 4 Distance map 
measured on a curved 
surface. Equidistant contours 
from the source located at the 
right hand are shown 
(Reproduced from [25]) 


Initialize d(xo) = 0 and mark x9 as processed; for all k 4 0 set d(x,) = 00 and 
mark x; as unprocessed. 
while there exist unprocessed vertices do 
Mark unprocessed neighbors of processed vertices as interface. 
for all interface vertices x and all incident triangles (x, x, x2) with x1,x2 4 
unprocessed do 
Update d(x) from d(x,) and d(x2). 
end for 
Mark interface vertex with the smallest value of d as processed. 
end while 


The general structure of fast marching closely resembles that of Dijkstra’s 
algorithm with the main difference lying in the update step. Unlike the graph case 
where shortest paths are restricted to pass through the graph edges, the continuous 
approximation allows paths passing anywhere in the simplicial complex. For that 
reason, the value of d(x) has to be computed from the values of the distance map 
at two other vertices forming a triangle with x. In order to guarantee consistency of 
the solution, all such triangles must have an acute angle at x. Obtuse triangles are 
split at a preprocessing stage by adding virtual connections to nonadjacent vertices. 

Given a triangle (x, x1, x2) with known values of d(x,) and d(x2), the goal of 
the update step is to compute d(x). The majority of fast marching algorithms do so 
by simulating the propagation of a planar wavefront in the triangle. The wavefront 
arrival time to x; and x2 is set to d(x,) and d(x2), from which the parameters of the 
wave source are estimated. Generally, there exist two solutions for d(x) consistent 
with the input, the smallest corresponding to the wavefront first arriving to x and 
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Fig. 5 Fast marching updates the triangle (x1, x2, x3) by estimating the planar wavefront direction 
n and origin p based on d, at x, and d at x2 and propagating it further to x3. d3 has two possible 
solutions: the one shown on the /eft is inconsistent, since d3 < d,, d. The solution on the right is 
consistent, since d3 > d, d). Geometrically, in order to be consistent, the update direction n has 
to form obtuse angles with the triangle edges (x3, x1) and (x3, x2) (Reproduced from [25]) 


then to x; and x2 and the largest corresponding to the inverse situation. In order 
to guarantee monotonicity of the solution, the largest solution is always chosen 
(Fig. 5). 

Computationally, fast marching has the O(N log N) complexity of Dijkstra, 
perhaps with a slightly larger constant. 


Parametric Surfaces 
For surfaces admitting a global parametrization x : U — R3, the eikonal equation 
can be expressed entirely in the parametrization domain as [104] 


VidG 'Vd =1, (25) 


where d(u) is the distance map in the parametrization domain, Vd is its gradient 
with respect to the standard basis in R?, and G are the coefficients of the first 
fundamental form in parametrization coordinates. The fast marching update step can 
be therefore performed on U. Moreover, since only G is involved in the equation, the 
knowledge of the actual vertex coordinates is not required. This property is useful 
when the surface is reconstructed from some indirect measurements, e.g., normals 
or gradients, as it allows to avoid surface reconstruction for metric computation. 


Parallel Marching 

The main disadvantage of all Dijkstra-type algorithms based on a heap structure 
in general and fast marching in particular is the fact that they are inherently 
sequential. Moreover, as the order in which the vertices are visited is unknown in 
advance, they typically suffer from inefficient access to memory. Working with well- 
structured parametric surfaces such as geometry images allows to circumvent these 
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Fig. 6 Raster scan grid traversals rotated by 45° (Reproduced from [116]) 


disadvantages by replacing the heap-based update by the regular raster scan update. 
Such family of algorithms is usually called parallel marching [117] or fast sweeping 
(111,121). 

In parallel marching, vertices of the geometry image are visited in a raster scan 
order, and for each vertex, the standard fast marching update is applied using already 
updated (causal) vertices as the supporting vertices for the following update. Four 
raster scans in alternating left-to-right top-to-bottom, right-to-left top-to-bottom, 
left-to-right bottom-to-top, and right-to-left bottom-to-top directions are applied (in 
practice, it is advantageous to rotate the scan directions by 45°, as shown in Fig. 6). 
For a Euclidean domain, such four scans are sufficient to consistently approximate 
the metric; for non-Euclidean shapes, several repetitions of the four scans are 
required. The algorithm stops when the distance map stops changing significantly 
from one repetition to another. The exact number of repetitions required depends on 
the metric and the parametrization, but is practically very small. 
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Parallel marching algorithms map well on modern vector and parallel architec- 
tures and in particular on graphics hardware [117]. 


Implicit Surfaces and Point Clouds 

Two-dimensional manifolds represented in the implicit form X = {®(x) = 0} Cc 
IR? can be approximated with arbitrary precision as the union of Euclidean balls of 
radius h > O around X, 


B,(X) = J BF). (26) 


xEX 


B,(X) is a three-dimensional Euclidean submanifold, which for h < 1/ max x2 has 
a smooth boundary. For every x, x’ € X, the shortest path in B,(X) is no longer 
than the corresponding shortest path on X. Mémoli and Sapiro [77] showed that as 
h — 0, shortest paths in B,(X) converge to those on X and the corresponding 
geodesic distances dg, (x)|xxx converge uniformly to dy. This result allows to 
cast the computation of a distance map on a curved two-dimensional space as the 
computation of a distance map on a three-dimensional Euclidean submanifold. The 
latter can be done using fast marching or parallel marching on orthogonal grid 
restricted to a narrow band around X [76]. 

A similar methodology can be used for the computation of distance maps on point 
clouds [76]. The union of Euclidean balls centered at each vertex of the cloud creates 
a three-dimensional Euclidean manifold, on which the distance map is computed 
using fast marching or parallel marching. 


Diffusion Distance 


The diffusion distance is expressed through the spectral decomposition of the 
Laplace—Beltrami operator, and its discretization involves the discretization of the 
Laplace—Beltrami operator and the computation of its eigenfunctions. 


Discretized Laplace-Beltrami Operator 


A discrete approximation of the Laplace—Beltrami on the mesh X has the following 
generic form 


1 
(Ag f)i = = y wij (fi = fi); (27) 
tj 


where f = (fi,..., fn) is a scalar function defined on the mesh Xx, wij are 
weights, and a; are normalization coefficients. In matrix notation, Eq. (27) can be 
written as 


Agf = ALE, (28) 
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where A = diag(a;) and L = diag (v4: wit) — (wij). 

Different discretizations of the Laplace—Beltrami operator lead to different 
choice of A and W. In general, it is common to distinguish between discrete and dis- 
cretized Laplace—Beltrami operator, the former being a combinatorial construction 
and the latter a discretization trying to preserve some of the properties (Li)—(Liv) of 
the continuous counterpart. In addition to these properties, it is important that the 
discrete Laplace—Beltrami operator converges to the continuous one, in the sense 
that the solution of the continuous heat equation with Ay converges to the discrete 
solution of the discrete heat equation with Ay as the number of samples grows to 
infinity. 

Purely combinatorial approximations such as the umbrella operator (w;; = 1 
if x; and x; are connected by an edge and zero otherwise) [120] and the Tutte 
Laplacian (wi; = ia , where dj is the valence of vertex x; ) [113] are not geometric, 
violate property (Liv), and do not converge to the continuous Laplace—Beltrami 
operator. One of the most widely used discretizations is the cotangent weight 
scheme [91] and its variants [79] (wi; = cota; +-cot B;; if x; and x; are connected, 
where @;; and §;; are the two angles opposite to the edge between vertices x; and x; 
in the two triangles sharing the edge and a; is proportional to the sum of the areas 
of the triangles sharing x; ). It preserves properties (Li)—(Liv) as well as satisfies the 
convergence property under certain mild conditions [116]. 


Computation of Eigenfunctions and Eigenvalues 
By solving the generalized eigendecomposition problem [67], 


A® = AL®, 


where ® is an N x (&k + 1) matrix whose columns are discretized eigenfunctions 
go, .--, Pg and A is the diagonal matrix of the corresponding eigenvalues Ag,... , Ax 
of the discretized Laplace—Beltrami operator are computed. ¢;; approximates the 
value of the /th eigenfunction at the point x;. 

A different approach to the computation of eigenfunction is based on the finite 
element method (FEM). Using the Green formula, the Laplace—Beltrami eigenvalue 
problem Ay = A¢ can be expressed in the weak form as 


(Ax@, a) r(x) = A(b, &) 1 5(x) (29) 


for any smooth a. Given a finite basis {a,...,a@x} spanning a subspace of L2(X), 
the solution @ can be expanded as $(x) © uja)(x) +--- + uxax (x). Substituting 
this expansion into (29) results in a system of equations 


K K 
Do uj (Axaj aK) 200) =A DY) uj (oj 0%)r209, k= 1... K, 


j=l j=l 
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which, in turn, is posed as a generalized eigenvalue problem 


Au = ABu. (30) 
(here A and B are K x K matrices with elements a,j; = (Aya;,ax)1,;y) and 
by; = (aj, %x)1,(x)). Solution of (30) gives eigenvalues A and eigenfunctions 


d = uj, +--+ uKaKx of Ay. 

As the basis, linear, quadratic, or cubic polynomials defined on the mesh can 
be used. Since the inner products are computed on the surface, the method is less 
sensitive to shape discretization compared to the direct approach based on the 
discretization of the Laplace—Beltrami operator. This is confirmed by numerical 
studies performed by Reuter et al. [94]. 


Discretization of Diffusion Distances 
Using the discretized eigenfunctions, a discrete diffusion kernel is approximated as 


k 
K(x;,x;) ® 2 K(A)) did ji, 


1=0 


and can be represented as an N x N matrix. The corresponding diffusion distance 
is approximated as 


1/2 


k 
dys (xi,x;) © (> K? (Ai) (Git — 6) 


1=1 


5 Invariant Shape Similarity 


Let us denote by X the space of all shapes equipped with some metric, ie., a 
point in X is a metric space (X,dy). Let 7 be a group of shape transformations, 
ie., a collection of operators t : X — X with the function composition. Two 
shapes differing by a transformation t € TJ are said to be equivalent up to T. 
The equivalence relation induces the quotient space X/T in which each point is 
an equivalence class of shapes that differ by a transformation in 7. A particular 
instance of 7 is the group of isometries, i.e., such transformations that acting on 
X leave dy unchanged. The exact structure of such the isometry group depends 
on the choice of the metric with which the shapes are equipped. For example, if 
the Euclidean metric dy = dg|yxx is used, the isometry group coincides with the 
group of Euclidean congruences (rotations, translations, and reflections). 

A function d : X x X —> R that associates a pair of shapes with a nonnegative 
scalar is called a distance or dissimilarity function. We will say that the dissimilarity 
d is T-invariant if it defines a metric on the quotient space X/7. In particular, 
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this means that d(X,1(X)) = O and d(t(X) x o(Y)) = d(X,Y) for every 
t,o € J and X,Y e€ X. In particular, for 7 being the isometry group, a 7- 
invariant dissimilarity is an isometry-invariant metric between shapes. The exact 
type of invariance depends on the structure of the isometry group and, hence, again 
on the choice of the metric with which the shapes are equipped. 

As a consequence from the metric axioms, an isometry-invariant dissimilarity 
d(X, Y) between two shapes X and Y equals to zeros if and only if X and Y are 
isometric. However, since exact isometry is an ideal rather than practical notion, it 
is desirable to extend this property to similar (almost isometric) rather than strictly 
equivalent (isometric) shapes. We will therefore require that (Ii) two €-isometric 
shapes X and Y satisfy d(X,Y) < cyje + by, and vice versa (Iii) if d(X, Y) < «, 
then X and Y are (c2¢+b2)-isometric, where c, C2, bj, and bz are some nonnegative 
constants. In what follows, we will focus on the construction of such dissimilarities 
and their approximation and show how different choices of the metric yield different 
classes of invariance. 


Rigid Similarity 


Equipping shapes with the restriction of the Euclidean metric in E allows to consider 
them as subsets of a bigger common metric space, E equipped with the standard 
Euclidean metric. We will therefore examine dissimilarity functions allowing to 
compare between two subsets of the same metric space. 


Hausdorff Distance 

For two sets X¥ and Y, a subset R C X x Y is said to be a correspondence between 

X and Y if (1) for every x € X there exists at least one y € Y such that (x,y) € R 

and (2) for every y € Y there exists at least one x € X such that (x, y) € R. We 

will denote by R(X, Y) the set of all possible correspondences between X and Y. 
Using the notion of correspondence, we can define the Hausdorff distance [58] 

between the two subsets of some metric space (Z, dz) as 


dé(xX,Y) = . 1 
H(X, Y) nce ee) (31) 


In other words, Hausdorff distance is the smallest nonnegative radius r for which 
B,(X) = U,ex B(x) © Y and B,(Y) C X, ie. 


d2(X,Y) = ind ind ; 2 
H(X, Y) = max oo a\>.y), aD z(x,y) (32) 


Hausdorff distance is a metric on the set of all compact nonempty sets in Z. 
However, it is not isometry invariant, i.e., for a nontrivial t € Iso(Z), generally 
de (X,1(X)) # 0. The isometry-invariant Hausdorff metric is constructed as the 
minimum of diz over all isometries in Z, 
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dels@y yy = min, dy (XY) (33) 


Elso( 


In the particular case of (Z, dz) being (E, dg), the isometry-invariant Hausdorff 
metric can be used to quantify similarity between rigid shapes measuring to which 
extent they are congruent (isometric in the Euclidean sense) to each other. The 
metric assumes the form 


dBlwoB@y y) = min dji(X, RY +t), (34) 
st 


where an orthogonal (rotation or, sometimes, rotation and reflection) matrix R and 
a translation vector ¢ € E are used to parametrize the Euclidean isometry group. 


Iterative Closest Point Algorithms 
Denoting by 


cpy (x) = min d (x, y) (35) 


the closest point to x in Y, the Euclidean isometry-invariant Hausdorff metric can 
be expressed as 


eames © 8 Y)= max | min max dg(x, RY +t), min max dg(y, RX + of 


Rywt xeEx Rit yey 


= min max } max ||x —c x)I]2, max || y — cp p— 
pin ax max x — ep 4 0°faemay LY —eP-i-n 0] 


(36) 


Such a formulation lends itself to numerical computation. A family of algorithms 
referred to as iterative closest point (ICP) [15, 36] first established the closest 
point correspondences between X and Y; once the correspondence is available, the 
Euclidean isometry (R, ft) minimizing max,ex ||x —cPpry+;(x)||2 and maxyey || y— 
CPr-1(y—1)(V)||2 is found and applied to Y. This, however, is likely to change the 
correspondence, so the process is repeated until convergence. For practical reasons, 
more robust variants of the Hausdorff distance are used [81]. 


Shape Distributions 

A disadvantage of the ICP algorithms is that the underlying optimization prob- 
lem becomes computationally intractable in high-dimensional spaces. A different 
approach for isometry-invariant comparison of rigid shapes, proposed by Osada et 
al. [85] and referred to as shape distribution, compares the distributions (histograms) 
of distances defined on the shape. Two isometric shapes obviously have identical 
shape distributions, which makes the approach isometry invariant. Shape distribu- 
tions can be computed in a space of any dimension, are computationally efficient, 
and are not limited to a specific metric. A notable disadvantage of shape distribution 
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distance is that it does not satisfy our axioms (Ii)—(lii), as there may be two non- 
isometric shapes with equal shape distributions; therefore, it is not a metric. 


Wasserstein Distances 
Let the sets X and Y be further equipped with measures zy and jLy, respectively. 
(It is required that supp(jry) = X and supp(wy ) = Y.) We will say that a measure 
jon X x Y is a coupling of wy and py if (i) w(X’ x Y) = wy(X’) and (ii) 
W(X x Y’) = py(Y’) for all Borel sets X’ C X and Y’ C Y. We will denote by 
M(x, [Ly ) the set of all possible couplings of zy and wy. The support supp(j2) of 
the measure jz is the minimum closed subset R C X x Y such that w~(R°) = 0. The 
support of each yp € M(jxx, ty) defines a correspondence; measure coupling can 
be therefore interpreted as a “soft” or “fuzzy” correspondence between two sets. 
The family of distances 


1 


d&, (ux.uy) = _ min (/, ABs yale »))" G7) 


HEM (Ly Hy) 


for 1 < p < ~, and 


Z . 
dW oo (Lx, Hy) = min max dz(x,y) (38) 
HEM (px Ly ) (x,y) €supp(.) 
for p = o is called the Wasserstein or Earth mover’s distances [97]. Wasserstein 
distances are metrics on the space of distributions (finite measures) on Z. For 
. . . . Z . . Z 
convenience, we will sometimes write dy, ,(X, Y) implying dy, ple x, by). 
Exactly like in the case of the Hausdorff distance, Wasserstein distances can be 
transformed into isometry-invariant metrics by considering the quotient with all 
isometries of Z, 


gz/o@ x, Y)= ani, dy p(X, “{Y)). (39) 


Wasserstein distances are intimately related to Monge—Kantorovich optimal 
transportation problems. Informally, if the measures zy and jy are interpreted 
as two ways of piling up a certain amount of dirt over the regions X and Y, 
respectively, and the cost of transporting dirt from point x to point y is quantified by 
ae (x, y), then the Wasserstein distance ae p &Xpresses the minimum cost of turning 
one pile into the other. On discrete domains, the Wasserstein distance can be cast as 
an optimal assignment problem and solved using the Hungarian algorithm or linear 
programming [97]. Several approximations have also been proposed in [60, 102]. 


Canonical Forms 


The Hausdorff distance allows comparing shapes equipped with the restricted 
Euclidean metric, i.e., considered as subsets of the Euclidean space. If other metrics 
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Fig. 7 Nearly isometric deformations of a shape (top row) and their canonical forms in R? (bottom 
row) 


are used, we have a more difficult problem of comparison of two different metric 
spaces. Elad and Kimmel [47,48] proposed an approximate solution to this problem, 
reducing it to the comparison of Euclidean subspaces by means of minimum- 
distortion embedding. Given a shape X with some metric d (e.g., geodesic or 
diffusion), it can be represented as a subset of the Euclidean space by means of an 
embedding y : X — IR’. If the embedding is isometric (dg|y(x)xy(x) 0° @ = @), the 
Euclidean representation (y(X), dz|y(x)xg(x)) called the canonical form of X can 
be used equivalently instead of (X,d) (Fig. 7). Given a Euclidean isometry i € E, 
if @ is isometric, then g 07 is also isometric. In other words, the canonical form 
is defined up to an isometry. In a more general setting, an arbitrary metric space 
(Z, dz) is used instead of E for the computation of the canonical form. 

The advantage of using canonical forms is that it brings the problem of shape 
comparison to the comparison of subsets of the Euclidean space, using, e.g., the 
Hausdorff distance. Given two shapes (X, d) and (Y, 5), their canonical forms y(X ) 
and w(Y) in Z are computed. In order to compensate for ambiguity in the definition 
of the canonical forms, an isometry-invariance distance between subsets of Z must 
be used, e.g., dt! mee (p(X), W(Y)). In the particular case of Euclidean canonical 


forms, ao (g(X), w(Y )) can be computed using ICP. 


Multidimensional Scaling 

Unfortunately, in most cases, there exists no isometric embedding of X into 
some predefined metric space. The right choice of Z can decrease the embedding 
distortion, but cannot make it zero [21, 114]. Instead, one can find an embedding 
with minimal distortion, 


min dis(g). 
p(X,d)—(Z,dz) (9) 
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In this case, dz|g(x)xyix) 9 g *& d, and thus the canonical form is only an 
approximate representation of the shape X in the space Z. 

In the discrete setting and Z = R”, given the discretized shape {x),..., xy } with 
the discretized metric d; j = d(x;,x;), the minimum-distortion embedding can be 
computed by solving the multidimensional scaling (MDS) problem [17,41], 


min max |d;; — ||z; —z; |||, 40 
{ct nZN} CR" if =... r ij — Nee — zl on) 


where z; = 9(x;). 

In practical applications, other norms (e.g., L2) are used in the MDS prob- 
lem (40). The MDS objective function is usually referred to as stress in MDS 
literature. For the Lz MDS problem (also known as least squares or LS-MDS), an 
efficient algorithm based on iterative majorization (commonly referred to as scaling 
by majorizing a complicated function or SMACOF) exists [44]. Denoting by Z the 
N X m matrix of the embedding coordinates of {x),...,xy}in R”, the SMACOF 
algorithm can be summarized as follows: 


Initialize embedding coordinate Z. 
for k = 1,2,--- do 
Perform multiplicative update 


1 
ZK) = y Bee ze, 


where B(Z) is an N x N matrix-valued function with elements 


dx(xi,xXj) > . 
moun) jz j and ley —z/l| #0. 


bi(Z) = 0 iA j and |i -zjll =0, 
= esi bik i= j. 


end for 


SMACOF iteration is equivalent to a weighted steepest descent with constant step 
size [32], but due to a special structure of the problem, it guarantees monotonous 
decrease of the stress function [17]. Other L, formulations can be solved using 
iteratively reweighted least-squares (IRLS) techniques [16]. Acceleration of con- 
vergence is possible using multiscale and multigrid methods [32] as well as vector 
extrapolation techniques [96]. 


Eigenmaps 

In the specific case when the shape is equipped with the diffusion distance (d = 
dx), the canonical form can be computed by observing the fact that the map 
Oy (x) = (e74 ho(x), e771 (x), ....) defined by the eigenvalues and eigenvec- 
tors of Laplace—Beltrami operator Ay satisfies dy;(x,x’) = ||®,(x) — ®;(x’)|lo. 


1888 A.M. Bronstein and M.M. Bronstein 


In other words, ®y, is an isometric embedding of (X,dy,) into an infinite- 
dimensional Euclidean space and can be thought of as an infinite-dimensional 
canonical form [13,72]. ®y, is termed Laplacian eigenmap [9] or diffusion 
map [40]. Another eigenmap given by Vy(x) = (477 bi(x), 22x), sig ), 
referred to as the global point signature (GPS) [98], is an isometric embedding of 
the commute time metric cy. 

Unlike Elad—Kimmel canonical forms computed by MDS, the eigenmap is 
uniquely defined (i.e., there are no degrees of freedom related to the isometry 
in the embedding space) if the Laplace—Beltrami operator has no eigenvalues 
of multiplicity greater than one. Otherwise, the ambiguity in the definition of 
the eigenmap is up to switching between the eigenfunction corresponding to the 
eigenvalues with multiplicity and changes in their signs. More ambiguities arise 
in cases of symmetric shapes [86]. In general, two eigenmaps may differ by a 
permutation of coordinates corresponding to simple eigenvalues or by a roto- 
reflection in the eigensubspaces corresponding to eigenvalues with multiplicities. 

For practical comparison of eigenmaps, a finite number k of eigenvectors is used, 
Oy, = (e~40' ho,...,e7**'@). The Euclidean distance on the eigenmap ®y, is 
thus a numerical approximation to the diffusion metric dy; using k eigenfunctions 
of the Laplace—Beltrami operator (similarly, Wy approximates the commute time). 
For small &, eigenmaps can be compared using ICP. The problem of coordinate 
permutations must be addressed if eigenvalues of multiplicity greater than one are 
present. Such an approach is impractical fork >> 1. 

As an alternative, Rustamov [98] proposed using shape distributions to compare 
eigenmaps. This method overcomes the aforementioned problem, but lacks the 
metric properties of a true isometry-invariant metric. 


Gromov-Hausdorff Distance 


The source of inaccuracy of Elad—Kimmel canonical forms is that it is generally 
impossible to select a common metric space (Z,dz) in which the geometry of 
any shape can be accurately represented. However, for given two shapes X and 
Y, the space (Z,dz) can be selected in such a way that (X,d) and (Y,6) can 
be isometrically embedded into it, the simplest example being the disjoint union 
Z = X UY of X and Y, with the metric dz|yxy = d and dzlyxy = 6. dz|xxy is 
defined to minimize the Hausdorff distance between X and Y in (Z, dz), resulting 
in a distance, 


dou(X, Y) = inf dy (X, Y), (41) 
Z 


called the Gromov—Hausdorff distance. The Gromov—Hausdorff distance was first 
proposed by Gromov [55] as a distance between metric spaces and a generalization 
of the Hausdorff distance and brought into shape recognition by Mémoli and 
Sapiro [78]. 
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The Gromov—Hausdorff distance satisfies axioms (Ii)—(Iii) with c; = cp = 2 and 
b, = by = 0, such that dgy(X, Y) = O if and only if X and Y are isometric. More 
generally, if dgy(X, Y) < €, then X and Y are 2¢-isometric, and, conversely, if X 
and Y are €-isometric, then dgy(X, Y) < 2e [34]. 

The Gromov—Hausdorff distance is a generic distance between metric spaces 
and, in particular, can be used to measure similarity between subsets of the 
Euclidean metric space, (X, dg|yxx) and (Y, dglyxy). In [75], Mémoli showed that 
the Gromov—Hausdorff distance in the Euclidean space is equivalent to the ICP 
distance, 


c+ dy! (X,Y) < don((X, del xxx), (Y, delyxy)) < dy" (x, ¥), 


in the sense of equivalence of metrics (c > 0 is a constant). (Metric equivalence 
should not be confused with equality: for example, L; and Lz metrics are equivalent 
but not equal.) Consequently, (1) if a ee Y) < e, then (X,dglxyxx) and 
(Y,dzlyxy) are 2¢-isometric; and (2) if (X,dzlxxx) and (Y,dzlyxy) are ¢€- 
isometric, then ma eg Y) <cdJe. 

Using the Gromov—Hausdorff distance to compare shapes equipped with dif- 
fusion metric allows to benefit from the advantage of the diffusion metric over 
geodesic one, such as lesser sensitivity to topological noise [28]. 


Generalized Multidimensional Scaling 
For compact metric spaces, the Gromov—Hausdorff distance can also be expressed 
as 


dou(X, Y) = ; inf dis (C), (42) 


where the infimum is taken over all correspondence C and dis(C). The two 
expressions (41) and (42) are equivalent [34]. 

The advantage of this formulation is that it allows to reduce the computation 
of the Gromov—Hausdorff distance to finding a minimum-distortion embedding, 
similarly to the computation of canonical forms by means of MDS. In the discrete 
setting, given two triangular meshes X andY representing the shapes X, Y, let us fix 
two sufficiently dense finite samplings P = {p1,..., Pm} and Q = {q1,...,qn} of 
: and Y, respectively. A discrete correspondence between the shapes is defined as 


= (P x QO’) U(O x P’), where P’ = (Tisnieabay and Q’ = Cee 

are some (different) sets of samples on X and Y corresponding to Q and P, 

respectively. One can represent C as u union of the graphs of two discrete 

functions g : P > Y and w : QO — X, parametrizing the class of all discrete 
correspondences. 

Given two sets P and P’ on X , we can construct an m Xx n distance matrix 

D(P, P’), whose elements are the distances dy (vi. 25) (e.g., geodesic or diffu- 


sion). In these terms, the distortion of the correspondence C can be written as 
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ais(C) = | ( PPP), PL. >) _ Be 2") DQ", ay 
D(P, P')" D(P’,P)} \D(Q',Q)" D(Q. Q) 


where ||- || is some norm on the space of (m + n) x (m +n) matrices. The 
selection of the infinity norm || D||oo = max;,; |dj;| is consistent with the Gromov— 
Hausdorff distance; however, in practice more robust norms like the Frobenius norm 
|| D || = trace(DD") are often preferable (see [23, 74, 78] for discussions on the 
regularization of the infinity norm in the Gromov—Hausdorff framework by other /,, 
norms). 

The discretization of dis (C) leads directly to a discretized approximation of the 
Gromov—Hausdorff distance between shapes, which can be expressed as 


a im 1 
dou(X, Y) := = min dis (C). 
ie) P’,O/ 


Note that only P’ and Q’ participate as continuous minimization variables, while 
P and Q are constants (given samples on the respective shapes). The above 
minimization problem is solved using a numerical procedure resembling MDS, first 
introduced in [23,24] under the name generalized MDS (GMDS). 

We use barycentric coordinates to represent points on X and Y. In these 
coordinates, a point p; lying in a triangle 7; on X is represented as a convex 
combination of the triangle vertices (corresponding to the indices t}, t?, and t?) 
with the weights u; = (u!,u2,u3)". We will denote by T = (ti,...,tm)! the 
vector of triangle indices and by U = (u,,..., um) the 3 x m matrix of coordinates 
corresponding to the sampling P. Similarly, the samplings P’, QO, and Q’ are 
represented as (T’, U’), (S,V), and (S’, V’). For the sake of notation simplicity, 
we are going to use these interchangeably. 

It was shown in [26] that a first-order approximation of a geodesic distance 
between p/ and Pi on X can be expressed as the quadratic form 


ia, P) Dia, P) Dag? P) 
D;j(P’, P’) = uit | D se P) Dy. 2(P. P) Dz AP, P) iui. 
Diy (2, P) D, af P) D; a(P, P) 
Gti Gp tj Gt 


Other distance terms are expressed similarly. Using tensor notation, we can write 
dis (C) © ||(U,U')Dg(T, T')(U, U') — V,V)Dg(S, SV. V9 |, 


where Dy (7, T’) is a rank four tensor whose ij -th elements are defined as the 3 x 3 
distance matrices above, and Dy (S, S’) is defined in a similar way. 

The resulting objective function dis(C) is a fourth-order polynomial 
with respect to the continuous coordinates U’,V’, also depending on the 
discrete index variables T’ and S’. However, when all indices and all 
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coordinate vectors except one, say, Uy, are fixed, the function becomes convex 


and quadratic with respect to uj. A closed-form minimizer of dis (u;) is 
found under the constraints uv} > 0 and ui! + ul? + ul = 1, guaran- 
teeing that the point p} remains within the triangle t/. The GMDS min- 
imization algorithm proceeds iteratively by selecting ui or vj correspond- 
ing to the largest gradient of the objective function, updating it accord- 
ing to the closed-form minimizer, and updating the corresponding triangle 
index to a neighboring one in case the solution is found on the boundary 
of the triangle. The reader is referred to [26] for further implementation 
details. 


Graph-Based Methods 


The minimum-distortion correspondence problem can be formulated as a binary 
labeling problem with uniqueness constraints [110] in a graph with vertices defined 
as pairs of points and edges defined as quadruplets. Let V = {(x,y) : x € 
X,y € Y} = X x be the set of pairs of points from X and Y, and let 
E = {((x, y), x’, y)) € V x Vand (x,y) 4 (x’, y’)}. A correspondence C C 
X x Y can be represented as binary labeling u € {0,1}” of the graph (V, €), as 
follows: ux, = 1 iff (x,y) € C and 0 otherwise. When using L» distortions, the 
correspondence problem can be reformulated as 


min, YP wayyy rider) = dry. y)P 
MOI xy) YEE 


st.) xy <1 Wx EX; Douy <1 VyeY. (43) 
y x 


In general, optimization of this energy is NP-hard [53]. One possible 
approximation of (43) is by relaxing the labels to be in [0,1]. This for- 
mulation leads to a non-convex quadratic program with linear constraints 
[46, 74]. Alternatively, instead of minimizing directly the energy (43), it is 
possible to maximize a lower bound on it by solving the dual to the linear 
programming (LP) relaxation of (43), a technique known as dual decompo- 
sition [110]. This approaches demonstrate good global convergence behavior 
[65]. 


Probabilistic Gromov—Hausdorff Distance 

The Gromov—Hausdorff framework can be extended to a setting in which pairwise 
distances are replaced by distributions of distances, modeling the intra-class 
variability shapes (e.g., the fact that different humans have legs of different length) 
[115]. The pairwise metric difference terms in the correspondence distortion are 
replaced by probabilities, and the problem is posed as likelihood maximization. 
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Gromov-Wasserstein Distances 
The same way as the Gromov—Hausdorff extends the Hausdorff distance by taking 


a minimum over all possible metric spaces, dgy = ming, di , an extension for the 
Wasserstein distance of the form 


daw. p(X, Y) = min dy p(X, Y) (44) 
Z 


al 
Pp 
min min d? (x, y)du(x, : 
dz vain. ( XxY a Yd ») 


referred to as Gromov—Wasserstein distance, was proposed by Mémoli [74]. Here, 
it is assumed that X and Y are metric measure spaces with metrics dy,dy and 
measures jLy, jy. The analogy between the Gromov—Hausdorff and the Gromov— 
Wasserstein distances is very close: the Hausdorff distance is a distance between 
subsets of a metric measure space, and the Gromov—Hausdorff distance is a distance 
between metric spaces. The Wasserstein distance is a distance between subsets of 
a metric space, and the Gromov—Wasserstein distance is a distance between metric 
measure spaces. 


Numerical Computation 
In [74], Mémoli showed that (44) can be alternatively formulated as 


dgw,p(X, Y) = (45) 


i 
P 


tn (/, I Jdxx.x") — dy(y, y Paes. yds») 
XxY JXxY 


HEM (Ly Hy) 


This formulation has an advantage in numerical implementation. Given discrete 
surfaces {x;,...,xv} and {y1,..., yy} with discretized metrics dy (x;, xj’), 
dy(y;, yj’) and measures y(x;),My(y;) (for i,i = 1,...,N and jf = 
1,...,M), problem (45) can be posed as an optimization problem with NM 
variables and N + M linear constraints: 


N M 
ming Yd) magmas ldx Ga. xv) — dv, yl? 


ij/=1 j,j/=1 


S.t. [iy € [0, 1] 


N 
YO Bi = py(y;) 
i=l 
M 


ie hij = [Lx (x;). 


j=l 
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Shape DNA 


Reuter et al. [95] proposed using the Laplace—Beltrami spectrum (i.e., eigenvalues 
Ao, A1,... of Ay) as shape descriptors, referred to as shape DNA. Laplace— 
Beltrami spectrum is isometry invariant; however, there may exist shapes which 
are isospectral (have equal eigenvalues) but non-isometric. This fact was first 
conjectured by Kac [62] and shown by example in [54]. Thus, the equivalence class 
of isospectral shapes to which the shape DNA approach is invariant is wider than the 
class of isometries. The exact relations between these classes are currently unknown. 


6 Partial Similarity 


In many situations, it happens that while two objects are not similar, some of 
their parts are. Such a situation is common, for example, in the face recognition 
application, where the quality of facial images (or surfaces in the case of 3D face 
recognition) can be degraded by acquisition imperfections, occlusions, and the 
presence of facial hair. Semantically, we can say that two objects are partially similar 
if they have significant similar parts. If one is able to detect such parts, the degree 
of partial similarity can be evaluated. 

We define a part of a shape (X, dy) simply as its subset X¥’ C X equipped with 
the restricted metric dy|y’xx’. According to this definition, every part of a shape is 
also a shape. We will denote by ©(X) C 2* the set of all admissible parts, satisfying 
(1) S(X) is nonempty; (2) U(X) is closed under complement, ie., if X¥’ € U(X), 
then X¥ \ X’ € X(X); and (3) N(X) is closed under countable unions, i.e., any 
countable union of parts from “(X) is also an admissible part in U(X). Formally, 
the set of all parts of X is a o-algebra. An equivalent representation of a part is by 
means of a binary indicator function, p : X — {0, 1}, assuming the value of one for 
each x € X’ and zero otherwise. We will see the utility of such a definition in the 
sequel. 


Significance 


The significance of a part is a function on &(X) assigning each part a number 
quantifying its “importance.” We denote significance by o and demand that (1) o 
is nonnegative; (2) o(@) = 0; and (3) o is countably additive, ie., 0 (U; X/) = 
re (xX; ) for every countable union of parts in &(X). Formally, significance is 
a finite measure on X. As in the case of similarity, the notion of significance is 
application dependent. The most straightforward way to define significance is by 
identifying it with the area 


a(X') = [a 
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or the normalized area 


/ da 
TX 


da 
xX 


o(X’) = 


of the part. However, such a definition might deem equally important a large flat 
region and a region rich in features if they have the same area, while it is clear that 
the latter one would usually be more informative. A better approach is to interpret 
significance as the amount of information about the entire shape contained in its part, 
quantified, e.g., as the ability to discriminate the shape from a given corpus of other 
shapes given only its part. Such a definition leads to a weighted area measure, where 
the weighting reflects the discriminativity density of each point and is constructed 
similarly to the term frequency-inverse document frequency (TF-IDF) weighting 
commonly used in text retrieval [2]. 


Regularity 


Another quantity characterizing the importance of a part is its regularity, which 
we model as a scalar function p : &(X) — R [18, 19]. In general, we would like 
the part to be simple, i.e., if two parts contain the same amount of information 
(are equally significant), we would prefer the simpler one, following Ockham’s 
pluralitas non est ponenda sine necessitate principle. What is exactly meant by 
“regular” and “simple” is again application dependent. In many applications, an 
acceptable definition of regularity is the deviation of a shape from some perfectly 
regular one. For example, in image processing and computer vision, regularity is 
commonly expressed using the shape factor 


or the ratio between the area of X’ and the squared length of its boundary. Because 
of the isoperimetric inequality in the plane, this ratio is always less or equal to one, 
with the equality achieved by a circle, which is arguably a very regular shape. Shape 
factor can be readily extended to non-Euclidean shapes, where, however, there is no 
straightforward analogy of the isoperimetric inequality. Consequently, two equally 
regular shapes might have completely different topology, e.g., one might have 
numerous disconnected components, while the other having only one (Fig. 8). 

A remedy can be in regarding regularity as a purely topological property, count- 
ing, for example, the number of disconnected components of a part. Topological 
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Fig. 8 Large shape factor 
does not necessarily imply 
regularity in non-Euclidean 
shapes. Here, the upper body 
of the dog and the four legs 
have the same area and the 
same boundary length (red 
contours) and, hence, the 
same shape factor. However, 
the upper body is arguably 
more regular than the four 
disconnected legs 
(Reproduced from [26]) 


regularity can be expressed in terms of the Euler characteristic, which using the 
Gauss—Bonnet identity becomes 


p(X’) = 2nyx(X"') = I, Kda+t+ Pee 


where K is the Gaussian curvature of X and kg is the geodesic curvature of 0X’. 


Partial Similarity Criterion 


In this terminology, the problem of partial similarity of two shapes X and Y can be 
thought finding two parts X’ € }(X) and Y’ € X(Y) simultaneously maximizing 
regularity, significance, and similarity. Since a part of a shape is also a shape, the 
latter can be quantified using any shape similarity (Since we use dissimilarity, 
we will maximize —d(X’, Y’).) criterion appropriate for the application, e.g., the 
Gromov—Hausdorff distance. This can be written as the following multi-criterion 
optimization problem [18, 19,27] 


X’), p(Y’),0(X’), oY"), —d(X', Y’)), 
Benue ), p(Y"),0(X),0(¥"), —d( )) 
Y/EX(Y) 


where maximum is understood as a point in the criterion space, such that no other 
point has all the criteria larger simultaneously. Such a maximum is said to be 
Pareto efficient and is not unique. The solution of this multi-criterion maximization 
problem can be interpreted as a set-valued partial similarity criterion. Since such 
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criteria are not mutually comparable, the problem should be converted into a scalar 
maximization problem 


max A,(p(X') + p(Y")) + As(a(X’) + o(¥’)) — d(X', YY"), (46) 
X/ED(X) 
Y’eX(Y) 


where A, and A, are positive scalars reflecting the tradeoff between regularity, 
significance, and dissimilarity. 


Computational Considerations 


Direct solution of problem (46) involves searching over the space of all parts of X 
and Y, which has combinatorial complexity. However, the problem can be relaxed 
to maximization in continuous variables if binary parts are allowed to be fuzzy. 
Formally, a fuzzy part is obtained by letting the binary indicator functions assume 
values on the interval [0, 1]. Such functions are called membership functions in the 
fuzzy set theory terminology. The optimization problem becomes [28] 


max A,(p(p) + p(q)) + As(a(p) + 0(9)) — d(p.9), 
p:X [0,1] 


q:Y>[0.1] 


where p(p),o(p) and d(p, q) are the fuzzy counterparts of the regularity, signifi- 
cance, and dissimilarity terms. The significance of a fuzzy part p is simply 


a() = | pd 


The regularity term is somewhat more involved as it involves integration along the 
part boundary, which does not exist in case of a fuzzy part. However, the following 
relaxation 1s available [35] 

An / pda 


i lien !) aa) 


with 6 being the Dirac delta function. This fuzzy version of the shape factor 
converges to the original definition when p approaches a binary indicator function. 
The dissimilarity term needs to be modified to involve the membership function. The 
most straightforward way to do so is by defining a weighted dissimilarity between 
the entire shapes X and Y with p and q serving as the weights. For example, using 
p(x)da(x) and q(y)da(y) as the respective measures on X and Y, the Wasserstein 
distance incorporates the weights in a natural way. 


P(p) = 
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7 Self-Similarity and Symmetry 


An important particular case of shape similarity is the similarity of shape with itself, 
which is commonly referred to as symmetry. The latter notion is intimately related 
with that of invariance. 


Rigid Symmetry 


Computation of exact and approximate symmetries has been extensively studied in 
the Euclidean sense [3,6,83, 118]. A shape X is said to be symmetric if there exists a 
nontrivial Euclidean isometry f € Iso(R*) to which it is invariant, i.e., f(X) = X. 
Such an isometry is called a symmetry of X. True symmetries, like true isometries, 
are a mere idealization not existing in practice. In real applications, we might still 
find approximate symmetries. The degree of asymmetry of a Euclidean isometry f 
can be quantified as a distance between X and f(X) in R?, e.g., 


asym(f) = dF (X, f(X)). 


Intrinsic Symmetry 


A symmetry f restricted to X defines a self-isometry of X, i.e., fly € Iso(X). 
Therefore, an alternative definition of an approximate symmetry could be an e- 
isometry, with the distortion quantifying the degree of asymmetry. Such a definition 
requires approximate symmetries to be automorphisms of X, yet its main advantage 
is the fact that it can be extended beyond the Euclidean case (Fig.9). In fact, 
identifying the symmetry group with the isometry group Iso(X, dy) of the shape 
X with some intrinsic (e.g., geodesic or diffusion) metric dy, a nonrigid equivalent 
of symmetries is defined, while setting dy = dglyxyx the standard Euclidean 
symmetries are obtained [92]. Approximate symmetries with respect to any metric 
can be computed as local minima of the distortion function in embedding X into 
itself. Computationally, the process can be carried out using GMDS. 


Spectral Symmetry 


An alternative to this potentially heavy computation is due to Ovsjanikov et al. [87] 
and is based on the elegant observation that for any simple (A simple eigenfunction 
is one corresponding an eigenvalues with multiplicity one.) eigenfunction ¢; of the 
Laplace—Beltrami operator, a reflection symmetry f satisfies ¢; o f = +¢;. This 
allows parametrize reflection symmetries by sign sequences s = {8),82,...}, Si € 
{+1}, such that ¢; o f = 5;9;. 
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6 


Extrinsic symmetry Intrinsic symmetry 
de = de 0 (9x g) dx = dx 0 (9x g) 


Fig. 9 Symmetry defined as a metric-preserving automorphism (self-isometry) of X allows 
extending the standard notion of Euclidean symmetry to nonrigid shapes (Reproduced from [92]) 


Given a sign sequence, the eigenmap ®,(x) = {say u(x), shy! bo(x), nee 
is defined. Symmetries are detected by evaluating the asymmetry 


asym(s) = max min || ®,(x’) — ®(x)|| 
xEX x/EX 


of different sign sequences and keeping those having asym < e. The symmetry 
itself corresponding to a sequence s is recovered as 


f(x) = arg min |®(x') — ©(8) |, 


and is an € self-isometry of X in the sense of the commute time metric. While 
it can be made relatively computationally simple, this method is limited to global 
reflection symmetries only. 


Partial Symmetry 


In many cases, a shape does not have symmetries as a whole, yet possesses parts 
that are symmetric. Adopting the notion of partial similarity defined in Sect. 6, 
one can think of a part X’ C X and a partial symmetry f : X' x X’ as of 
a Pareto-efficient tradeoff between asymmetry asym(/), part significance 0 (X’), 
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and regularity o(X’). Partial symmetries are found similarly to the computation of 
partial similarity of two distinct shapes. 


Repeating Structure 


Another important particular case of self-similarity is repeating regular struc- 
ture. Shapes possessing regular structure can be divided into self-similar patches 
(structural elements) forming some regular patterns, e.g., a grid. State-of-the-art 
methods [89, 108? ] can detect structured repetitions in extrinsic geometry if the 
Euclidean transformations between repeated patches exhibit group-like behavior. 
In case of nonrigid and deformable shapes, however, the problem is challenging 
since no apparent structure is visible to simple Euclidean probes in the absence of 
repetitive Euclidean transformations to describe the shape. A general solution for 
the detection of intrinsic regular structure is still missing, though particular cases 
have been recently addressed in [29]. 


8 Feature-Based Methods 


Another class of methods, referred to as feature based, uses local information to 
describe the shape, perform matching, or compute similarity. The popularity of these 
methods has increased following the success of the scale-invariant feature transform 
(SIFT) [70] and similar algorithms [8,71] in image analysis and computer vision 
application. 


Feature Descriptors 


In essence, feature-based methods try to represent the shape as a collection 
of local feature descriptors. This is typically done in two steps first, selecting 
robust and representative points (feature detection) and computing the local shape 
representation at these points (feature description). 


Feature Detection 

One of the main requirements on a feature detector is that the points it selects are (1) 
repeatable, i.e., in two instances of a shape, ideally the same set of corresponding 
points is detected, and (2) informative, i.e., the information contained in these points 
is sufficient to, e.g., distinguish the shape from others. 

In the most trivial case, no feature detection is performed and the feature 
descriptor is computed at all the points of the shape or at some regularly sampled 
subset thereof. The descriptor in this case is usually termed dense. Dense descriptors 
bypass the problem of repeatability at the price of increased computational 
cost and potentially introducing many unimportant points that clutter the shape 
representation. 
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Many geometric feature detection paradigms come from the image analysis 
community, such as finding points with high derivatives (e.g., the Harris operator 
(30, 52,57]) or local maxima in a scale space (e.g., difference of Gaussians (DOG) 
[119] or local maxima of the heat kernel [49]). 


Feature Description 

Given a set of feature points (or, in the case of a dense descriptor, all the points on the 
shape), a local descriptor is then computed. An ideal feature descriptor should be (1) 
invariant under the class of transformations a shape can undergo and (2) informative. 
One of the most known feature descriptors is spin image [4,5, 61], describing 
the neighborhood of a point by fitting an oriented coordinate system at the point. 
Belongie and Malik introduced the shape context descriptor [11], describing the 
structure of the shape as relations between a point to the rest of the point. Given the 
coordinates of a point x on the shape, the shape context descriptor is constructed 
as a histogram of the direction vectors from x to the rest of the point, y — x. 
Typically, a log-polar histogram is used. Because of dependence on the embedding 
coordinates, such a descriptor is not deformation invariant. Other descriptors exist 
based on local patches [82], local moments [38] and volume descriptors [50], 
spherical harmonics [101], and contour and edge structures [64,88]. Zaharescu et al. 
[119] proposed using as a local descriptor the histogram of gradients of a function 
(e.g., Gaussian curvature) defined in a neighborhood of a point, similarly to the 
histogram of gradients (HOG) [43] and SIFT [70] techniques used in computer 
vision. 

Because considering local geometry, feature descriptors are usually not very 
susceptible to nonrigid deformations of the shape. Nevertheless, there exist several 
geometric descriptors which are invariant to isometric deformations by construction. 
Examples include descriptors based on histograms of local geodesic distances [27], 
conformal factors [12], and heat kernels [106], described in the following in more 
details. 


Heat Kernel Signatures 

Sun et al. [106] proposed the heat kernel signature (HKS), defined as the diagonal 
of the heat kernel. Given some fixed time values t),...,¢,, for each point x on the 
shape, the HKS is an n-dimensional descriptor vector 


P(x) = (Ki, (x, x),..., Kz, (*, x)). (47) 


The HKS descriptor is deformation invariant, captures local geometric information 
at multiple scales, is insensitive to topological noise, is informative (if the Laplace— 
Beltrami operator of a shape is non-degenerate, then any continuous map that 
preserves the HKS at every point must be an isometry), and is easily computed 
across different shape representations solving the eigenproblem described in sec- 
tion “Diffusion Distance.” 
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Scale-Invariant Heat Kernel Signatures 

A disadvantage of the HKS is its dependence on the global scale of the shape. If X 
is globally scaled by 8, the corresponding HKS is BP Kp, (x, x). In some cases, 
it is possible to remove this dependence by global normalization of the shape. A 
scale-invariant HKS (SI-HKS) based on Jocal normalization was proposed in [33]. 
By using a logarithmic scale-space t = a’, the scaling of X by # results in HKS 
amplitude scaling and shift by 2log, 8. This effect is undone by the following 
sequence of transformations, 


Daif (x) = (log Kyx(x, x) — log Kan (x, x),..., log Kam (x, x) 
—log Katm—i (x, x)), P(x) = |(F pait (x))(@1,..-,@n)|, (48) 


where F is the discrete Fourier transform and @,...,@, denotes a set of frequen- 
cies at which the transformed vector is sampled. Taking differences of logarithms 
removes the scaling constant, and the Fourier transform converts the scale-space 
shift into a complex phase, which is removed by taking the absolute value. 


Bags of Features 


One of the notable advantages of feature-based approaches is the possibility of 
representing a shape as a collection of primitive elements (“geometric words”) 
and using the well-developed methods from text search such as the bag of features 
(BOF) (or bag of words) paradigm [37, 103]. Such approaches are widely used in 
image retrieval and have been introduced more recently to shape analysis [29, 109]. 
The bag of features representation is usually compact and easy to store and compare, 
which makes such approaches suitable for large-scale shape retrieval. 

The construction of a bag of features is usually performed in a few steps, depicted 
in Fig. 10. First, the shape is represented as a collection of local feature descriptors 
(either dense or computed at a set of stable points following an optional stage of 
feature detection). Second, the descriptors are represented by geometric words from 
a geometric vocabulary using vector quantization. The geometric vocabulary is a set 
of representative descriptors, precomputed in advance. This way, each descriptor is 
replaced by the index of the closest geometric word in the vocabulary. Computing 
the histogram of the frequency of occurrence of geometric words gives the bag 
of features. Alternatively, a two-dimensional histogram of co-occurrences of pairs 
of geometric words (geometric expressions) can be used [29]. Shape similarity is 
computed as a distance between the corresponding bags of features. 


Combining Global and Local Information 


Another use of local descriptors is in combination with global (metric) information, 
in an extension of the Gromov—Hausdorff framework. Given two shapes X, Y with 
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Fig. 10 Feature-based shape analysis algorithm (Reproduced from [29]) 


metrics dy, dy and descriptors px, py, the quality of correspondence C C X xY is 
measured using global geometric distortion as well as local matching of descriptors, 


dis(C)= = sup_—_|dx (x, x’)—dy(y, y’)| +f sup ||px(x) — py(y)|l. 
(x.y). (x.y EC (x,y)EC 


where 6 > 0 is some parameter. This Loo formulation can be replaced by a more 
robust Lz version. As the descriptors, texture [105, 107] or geometric information 
[46, 115] can be used. 

The minimum-distortion correspondence can be found by an extension of the 
GMDS algorithm described in section “Generalized Multidimensional Scaling” 
[107] or graph labeling [105,110,115] described in section “Graph-Based Methods.” 
The probabilistic extension of the Gromov—Hausdorff distance can be applied to this 
formulation as well [115]. 


9 Conclusion 


In this chapter, the problem of invariant shape similarity was presented through 
the prism of metric geometry. It was shown that by representing shapes as metric 
spaces allows to reduce the similarity problem to isometry-invariant comparison 
of metric spaces. The particular choice of the metric results in different isometry 
groups and, hence, different invariance classes. The construction of Euclidean, 
geodesic, and diffusion metrics was presented, and their theoretical properties were 
highlighted in Sect.2. Based on these notions, different shape similarity criteria 
and distances were presented in Sect.5, fitting well under the metric umbrella. 
Computational aspects related to shape and metric discretization were discussed 
in Sects.3 and 4, and computation of full and partial similarity was discussed 
in Sects.5 and 6. In Sect. 8, feature-based methods were discussed. For further 
detailed discussion of these and related subjects, the reader is referred to the book 
[25]. 
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1 Introduction 
Image Analysis and Prior Knowledge 


Image segmentation is among the most studied problems in image understanding 
and computer vision. The goal of image segmentation is to partition the image plane 
into a set of meaningful regions. Here meaningful typically refers to a semantic 
partitioning where the computed regions correspond to individual objects in the 
observed scene. Unfortunately, generic purely low-level segmentation algorithms 
often do not provide the desired segmentation results, because the traditional low- 
level assumptions like intensity or texture homogeneity and strong edge contrast are 
not sufficient to separate objects in a scene. 

To overcome these limitations, researchers have proposed to impose prior 
knowledge into low-level segmentation methods. In the following, we will review 
methods which allow to impose knowledge about the shape of objects of interest 
into segmentation processes. 

In the literature there exist various definitions of the term shape, from the 
very broad notion of shape of Kendall [54] and Bookstein [5] where shape is 
whatever remains of an object when similarity transformations are factored out (i.e., 
a geometrically normalized version of a gray value image) to more specific notions 
of shape referring to the geometric outline of an object in 2D or 3D. In this work, we 
will adopt the latter view and refer to an object’s silhouette or boundary as its shape. 
Intentionally we will leave the exact mathematical definition until later, as different 
representations of geometry actually imply different definitions of the term shape. 

One can distinguish various kinds of shape knowledge: 


¢ Low-level shape priors which typically simply favor shorter boundary length, 1.e., 
curves with shorter boundary have lower shape energy, where boundary length 
can be measured in various ways [4, 6, 49, 53, 69]. 

e Mid-level shape priors which favor, for example, thin and elongated structures, 
thereby facilitating the segmentation of roads in satellite imagery or of blood 
vessels in medical imagery [44, 70, 78]. 

e High-level shape priors which favor similarity to previously observed shapes, 
such as hand shapes [22,36,50], silhouettes of humans [26,29], or medical organs 
like the heart, the prostate, the lungs, or the cerebellum [58, 82, 84, 99]. 


There exists a wealth of works on shape priors for image segmentation. It is 
beyond the scope of this article to provide a complete overview of existing work. 
Instead, we will present a range of representative works — with many of the examples 
taken from the author’s own work — and discuss their advantages and shortcomings. 
Some of these works are formulated in a probabilistic setting where the challenge is 
to infer the most likely shape given an image and a set of training shapes. Typically 
the segmentation is formulated as an optimization problem. 

One can distinguish two important challenges: 
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1. The modeling challenge: How do we formalize distances between shapes? What 
probability distributions do we impose? What energies should we minimize? 

2. The algorithmic challenge: How do we minimize the arising cost function? Are 
the computed solutions globally optimal? If they are not globally optimal, how 
sensitive are solutions with respect to the initialization? 


Explicit Versus Implicit Shape Representation 


A central question in the modeling of shape similarity is that of how to represent a 
shape. Typically one can distinguish between explicit and implicit representations. 
In the former case, the boundary of the shape is represented explicitly — in a 
spatially continuous setting, this could be a polygon or a spline curve. In a spatially 
discrete setting this could be a set of edgles (edge elements) forming a regular grid. 
Alternatively, shapes can be represented implicitly in the sense that one labels all 
points in space as being part of the interior or the exterior of the object. In the 
spatially continuous setting, the optimization of such implicit shape representations 
is solved by means of partial differential equations. Among the most popular 
representatives are the level set method [39, 72] or alternative convex relaxation 
techniques [11]. In the spatially discrete setting, implicit representations have 
become popular through the graph cut methods [7, 49]. More recently, researchers 
have also advocated hybrid representations where objects are represented both 
explicitly and implicitly [90]. Table | provides an overview of a few representative 
works on image segmentation based on explicit and implicit representations of 
shape. 

Figure | shows examples of shape representations using an explicit parametric 
representation by spline curves (spline control points are marked as black boxes), 
implicit representations by a signed distance function or a binary indicator function, 
and an explicit discrete representation (4th image). 

As we shall see in the following, the choice of shape representation has important 
consequences on the class of objects that can be modeled, the type of energy that 
can be minimized, and the optimality guarantees that can be obtained. Among the 
goals of this article is to put in contrast various shape representations and discuss 
their advantages and limitations. In general one observes that: 


Table 1 Shapes can be represented explicitly or implicitly, in a spatially continuous or a spatially 

discrete setting. More recently, researchers have adopted hybrid representations [90], where objects 

are represented both in terms of their interior (implicitly) and in terms of their boundary (explicitly) 
Spatially continuous Spatially discrete 

Explicit Polygons [22,102], splines _| Edgel labeling and dyn. progr. | Hybrid repres. and LP 
[3, 36, 53] [1, 74, 80, 87, 89] relaxation [90] 

Implicit Level set methods [39,72], Graph cut methods [6,49] 

convex relaxation [11,31] 
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Fig. 1 Examples of shape representations by means of a parametric spline curve (1st image), 
a signed distance function (2nd image), a binary indicator function (3rd image), and an explicit 
discrete representation (4th image) 


Implicit representations are easily generalized to shapes in arbitrary dimen- 
sion. Respective algorithms (level set methods, graph cuts, convex relaxation 
techniques) straightforwardly extend to three or more dimensions. Instead, 
the extension of explicit shape representations to higher dimensions is by no 
means straightforward: The notion of arc-length parameterization of curves does 
not extend to surfaces. Moreover, the discrete polynomial-time shortest-path 
algorithms [1, 85,89] which allow to optimally identify pairwise correspondence 
of points on either shape do not directly extend to minimal-surface algorithms. 
Implicit representations are easily generalized to arbitrary shape topology. Since 
the implicit representation merely relies on a labeling of space (as being inside 
or outside the object), the topology of the shape is not constrained. Both level 
set and graph cut algorithms can therefore easily handle objects of arbitrary 
topology. Instead, for spatially continuous parametric curves, modeling the 
transition from a single closed curve to a multiply connected object boundary 
requires sophisticated splitting and merging techniques [38,60,61,65]. Similarly, 
discrete polynomial-time algorithms are typically constrained to finding open 
[1, 20, 23] or closed curves [86, 89]. 

Explicit boundary representations allow to capture the notion of point correspon- 
dence [47, 85, 89]. The correspondence between points on either of two shapes 
and the underlying correspondence of semantic parts is of central importance 
to human notions of shape similarity. The determination of optimal point 
correspondences, however, is an important combinatorial challenge, especially 
in higher dimensions. 

For explicit representations, the modeling of shape similarity is often more 
straightforward and intuitive. For example, for two shapes parameterized as 
spline curves, the linear interpolation of these shapes also gives rise to a 
spline curve and often captures the human intuition of an intermediate shape. 
Instead, the linear interpolation of implicit representations is generally not 
straightforward: Convex combinations of binary-valued functions are no longer 
binary-valued. And convex combinations of signed distance functions are gener- 
ally no longer a signed distance function. Figure 2 shows examples of a linear 
interpolations of spline curves and a linear interpolations of signed distance 
functions. Note that the linear interpolation of signed distance functions may 
give rise to intermediate silhouettes of varying topology. 
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Fig. 2 The linear interpolation of spline-based curves (shown here along the first three eigen- 
modes of a shape distribution) gives rise to a families of intermediate shapes 


aA 


Fig. 3 This figure shows the linear interpolation of the signed distance functions associated with 
two human silhouettes. The interpolation gives rise to intermediate shapes and allows changes of 
the shape topology. Yet, the linear combination of two signed distance functions is generally no 
longer a signed distance function 


In the following, we will give an overview over some of the developments in the 
domain of shape priors for image segmentation. In Sect. 2, we will review a formula- 
tion of image segmentation by means of Bayesian inference which allows the fusion 
of input data and shape knowledge in a single energy minimization framework 
(Fig. 3). In Sect. 3, we will discuss a framework to impose statistical shape priors in 
a spatially continuous parametric representation. In Sect. 4, we discuss methods to 
impose statistical shape priors in level set based image segmentation. In Sect. 5, we 
discuss statistical models which allow to represent the temporal evolution of shapes 
and can serve as dynamical priors for image sequence segmentation. And lastly, 
in Sect. 6, we will present recent developments to impose elastic shape priors in a 
manner which allows to compute globally optimal shape-consistent segmentations 
in polynomial time. 


2 Image Segmentation via Bayesian Inference 


Over the last decades Bayesian inference has become an established paradigm to 
tackle data analysis problems — see [30, 105] for example. Given an input image 
I: Q — Rona domain Q C R’, a segmentation C of the image plane Q can be 
computed by maximizing the posterior probability: 
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PU |C) PC) 


PCat oa 


(1) 


where P(/ |C) denotes the data likelihood for a given segmentation C and P(C) 
denotes the prior probability which allows to impose knowledge about which 
segmentations are a priori more or less likely. 

Maximizing the posterior distribution can be performed equivalently by mini- 
mizing the negative logarithm of (1) which gives rise to an energy or cost function 
of the form 


E(C) = Eaata(C) + Eshape(C), (2) 


where Egata(C) = —log PU |C) and Eghape(C) = — log P(C) are typically referred 
to as data fidelity term and regularizer or shape prior. By maximizing the posterior, 
one aims at computing the most likely solution given data and prior. Of course there 
exist alternative strategies of either computing solutions corresponding to the mean 
of the distribution rather than its mode or of retaining the entire posterior distribution 
in order to propagate multiple hypotheses over time, as done, for example, in the 
context of particle filtering [3]. 

Over the years various data terms have been proposed. In the following, we will 
simply use a piecewise-constant approximation of the input intensity J [69]: 


k 
Eat) = I(x) — ;)dx, 3 
aaa(C) > fu ee 3) 


where the regions Q),...,@ x are pairwise disjoint regions separated by the 
boundary C and jz; denotes the average of J over the region Q2;: 


1 
b= [, Roae (4) 


More sophisticated data terms based on color likelihoods [8, 57, 103] or texture 
likelihoods [2, 30] are conceivable. 

A glance into the literature indicates that the most prominent image segmentation 
methods rely on a rather simple geometric shape prior Ehape which energetically 
favors shapes with shorter boundary length [4, 53, 69], a penalizer which — 
in a spatially discrete setting — dates back at least as far as the Ising model 
for ferromagnetism [52]. There are several reasons for the popularity of length 
constraints in image segmentation. Firstly, solid objects in our world indeed tend 
to be spatially compact. Secondly, such length constraints are mathematically well 
studied. They give rise to well-behaved models and algorithms — mean curvature 
motion in a continuous setting and low-order Markov random fields and submodular 
cost functions in the discrete setting. 
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Nevertheless, the preference for a shorter boundary is clearly a very simplistic 
shape prior. In many applications the user may have a more specific knowledge 
about what kinds of shapes are likely to arise in a given segmentation task. For 
example, in biology one may want to segment cells that all have a rather specific 
size and shape. In medical imaging one may want to segment organs that all have a 
rather unique shape — up to a certain variability — and preserve a specific spatial 
relationship with respect to other organs. In satellite imagery one may be most 
interested in segmenting thin and elongated roads, or in the analysis of traffic scenes 
from a driving vehicle, the predominant objects may be cars and pedestrians. In 
the following sections, we will discuss ways to impose such higher-level shape 
knowledge into image segmentation methods. 


3 Statistical Shape Priors for Parametric Shape 
Representations 


Among the most straightforward ways to represent a shape is to model its outline 
as a parametric curve. An example is a simple closed spline curve C € C*(S!, Q) of 
the form 


C(s) = )> pi Bi(s), (5) 


i=1 


where p; € R? denote a set of spline control points and B; a set of spline basis 
functions of degree k [19, 36, 43, 66]. In the special case of linear basis functions, 
we simply have a polygonal shape, used, for example, in [102]. With increasing 
number of control points, we obtain a more and more detailed shape representation 
— see Fig. 4. It shows one of the nice properties of parametric shape representations: 
The representation is quite compact in the sense that very detailed silhouettes can 
be represented by a few real-valued variables. 

Given a family of m shapes, each represented by a spline curve of a fixed number 
of n control points, we can think of these training shapes as a set {z,...,Zm} of 
control point vectors: 


Input image 20 points 40 points 70 points 100 points 


Fig. 4 Spline representation of a hand shape (/eft) with increasing resolution 
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Zi = (pit,--+, Pin) € R™, (6) 


where we assume that all control point vectors are normalized with respect to 
translation, rotation, and scaling [41]. 

With this contour representation, the image segmentation problem boils down to 
computing an optimal spline control point vector z € R” for a given image. The 
segmentation process can be constrained to familiar shapes by imposing a statistical 
shape prior computed from the set of training shapes. 


Linear Gaussian Shape Priors 


Among the most popular shape prior is based on the assumption that the training 
shapes are Gaussian distributed — see for example [22, 36,55]. There are several 
reasons for the popularity of Gaussian distributions. Firstly, according to the 
central limit theorem, the average of a large number of i.i.d. random variables is 
approximately Gaussian distributed — so if the observed variations of shape were 
created by independent processes, then one could expect the overall distribution 
to be approximately Gaussian. Secondly, the Gaussian distribution can be seen as 
a second-order approximation of the true distribution. And thirdly, the Gaussian 
distribution gives rise to a convex quadratic cost function that allows for easy 
minimization. 

In practice, the number of training shapes m is often much smaller than 
the number of dimensions 2n. Therefore, the estimated covariance matrix » is 
degenerate with many zero eigenvalues and thus not invertible. As introduced in 
[36], a regularized covariance matrix is given by 


Hr=h+ aL (-VV'), (7) 


where V is the matrix of eigenvectors of X. In this way, we replace all zero 
eigenvalues of the sample covariance matrix & by a constant A, €[0,A,], where 
A, denotes the smallest nonzero eigenvalue of &. (Note that the inverse ay of 
the regularized covariance matrix defined in (7) fundamentally differs from the 
pseudoinverse, the former scaling components in degenerate directions by Pg 
while the latter scaling them by 0.) In [68] it was shown that 2; can be computed 
from the true covariance matrix by minimizing the Kullback—Leibler divergence 
between the exact and the approximated distribution. Yet, since we do not have the 
exact covariance matrix but merely a sample covariance matrix, the reasoning for 
determining A suggested in [68] is not justified. 
The Gaussian shape prior is then given by 


1 1 aes 2 
PQ = j2xd_|'72 exp (-; (¢—2' ST! @-2) , (8) 


where z denotes the mean control point vector. 
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Based on the Gaussian shape prior, we can define a shape energy that is invariant 
to similarity transformations (translation, rotation, and scaling) by 


E shape(Z) = —logP (2) , (9) 


where Z is the shape vector upon similarity alignment with respect to the training 
shapes: 


R(z— 2) 


—— 10 
|R (z — Zo)| se 


t= 


where the optimal translation zp and rotation R can be written as functions of z [36]. 
As a consequence, we can minimize the overall energy 


E(z) = Euaa(C(z)) +. Eshape(z) (11) 


using gradient descent in z. For details on the numerical minimization, we refer to 
[25, 36]. 

Figure 5 shows several intermediate steps in a gradient descent evolution on the 
energy (2) combining the piecewise constant intensity model (3) with a Gaussian 
shape prior constructed from a set of sample hand shapes. Note how the similarity- 


a aL 


Initial curve Step 1 Step 2 Step 3 


NE 


Step 4 Step 5 Final Training shapes 


Fig. 5 Evolution of a parametric spline curve during gradient descent on the energy (2) combining 
the piecewise constant intensity model (3) with a Gaussian shape prior constructed from a set of 
sample hand shapes (lower right). Note that the shape prior is by construction invariant to similarity 
transformations. As a consequence, the contour easily undergoes translation, rotation, and scaling 
as these do not affect the energy 
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Fig. 6 Gradient descent evolution of a parametric curve from initial to final with similarity- 
invariant shape prior. The statistical shape prior permits a reconstruction of the hand silhouette 
in places where it is occluded 


invariant shape prior (9) constrains the evolving contour to hand-like shapes without 
constraining its translation, rotation, or scaling. 

Figure 6 shows the gradient descent evolution with the same shape prior for an 
input image of a partially occluded hand. Here the missing part of the silhouette is 
recovered through the statistical shape prior. These evolutions demonstrate that the 
curve converges to the correct segmentation over rather large spatial distance, an 
aspect which is characteristic for region-based cost functions like (3). 


Nonlinear Statistical Shape Priors 


The shape prior (9) was based on the assumption that the training shapes are 
Gaussian distributed. For collections of real-world shapes, this is generally not the 
case. For example, the various silhouettes of a rigid 3D object obviously form a 
three-dimensional manifold (given that there are only three degrees of freedom 
in the observation process). Similarly, the various silhouettes of a walking person 
essentially correspond to a one-dimensional manifold (up to small fluctuations). 
Furthermore, the manifold of shapes representing deformable objects like human 
persons are typically very low-dimensional, given that the observed 3D structure 
only has a small number of joints. 

Rather than learning the underlying low-dimensional representation (using 
principal surfaces or other manifold learning techniques), we can simply estimate 
arbitrary shape distributions by reverting to nonlinear density estimators — nonlinear 
in the sense that the permissible shapes are not simply given by a weighted sum 
of eigenmodes. Classical approaches for estimating nonlinear distributions are the 
Gaussian mixture model or the Parzen—Rosenblatt kernel density estimator — see 
Sect. 4. 

An alternative technique is to adapt recent kernel learning methods to the 
problem of density estimation [28]. To this end, we approximate the training shapes 
by a Gaussian distribution, not in the input space but rather upon transformation 
w : R™” — Y to some generally higher-dimensional feature space Y: 


1 
Py (2) exp (—5 (2) ~ vo)! Ey! HO) — Wo)) (12) 
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As before, we can define the corresponding shape energy as 
E(z) = —logPy (@), (13) 


with Z being the similarity-normalized shape given in (10). Here wo and Xy, denote 
the mean and covariance matrix computed for the transformed shapes: 


it 1 
Yo Fr evG@), Ly = a (() — Wo) (Wei) — Yo)", (14) 


i=1 i=1 


where &,, is again regularized as in (7). 

As shown in [28], the energy E(z) in (13) can be evaluated without explicitly 
specifying the nonlinear transformation w. It suffices to define the corresponding 
Mercer kernel [24, 67]: 


k(x, y) = (W(x). WQ)), Vx.y ER”, (15) 


representing the scalar product of pairs of transformed points w(x) and y(y). In the 
following, we simply chose a Gaussian kernel function of width o: 


1 Ilx — yl? 


It was shown in [28] that the resulting energy can be seen as a generalization of the 
classical Parzen—Rosenblatt estimators. In particular, the Gaussian distribution in 
feature space Y is fundamentally different from the previously presented Gaussian 
distribution in the input space R”’. Figure 7 shows the level lines of constant 
shape energy computed from a set of left- and right-hand silhouettes, displayed 


SH i 


Aligned contours Simple gaussian Mixture model Feature space 
gaussian 


Fig. 7 Model comparison. Density estimates for a set of left (e) and right (++) hands, projected 
onto the first two principal components. From Jeft to right: aligned contours, simple Gaussian, 
mixture of Gaussians, and Gaussian in feature space (13). In contrast to the mixture model, the 
Gaussian in feature space does not require an iterative (sometimes suboptimal) fitting procedure 
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Fig. 8 ‘Tracking a familiar object over a long image sequence with a nonlinear statistical shape 
prior. A single shape prior constructed from a set of sample silhouettes allows the emergence of a 
multitude of familiar shapes, permitting the segmentation process to cope with background clutter 
and partial occlusions 


in a projection onto the first two eigenmodes of the distribution. While the linear 
Gaussian model gives rise to elliptical level lines, the Gaussian mixture and the 
nonlinear Gaussian allow for more general non-elliptical level lines. In contrast to 
the mixture model, however, the nonlinear Gaussian does not require an iterative 
parameter estimation process, nor does it require or assume a specific number of 
Gaussians. 

Figure 8 shows screenshots of contours computed for an image sequence 
by gradient descent on the energy (11) with the nonlinear shape energy (13) 
computed from a set of 100 training silhouettes. Throughout the entire 
sequence, the object of interest was occluded by an artificially introduced 
rectangle. Again, the shape prior allows to cope with spurious background 
clutter and to restore the missing parts of the object’s silhouette. Two- 
dimensional projections of the training data and evolving contour onto the 
first principal components, shown in Fig.9, demonstrate how the nonlinear 
shape energy constrains the evolving shape to remain close to the training 
shapes. 
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Fig. 9 Tracking sequence 
from Fig. 8 visualized. 
Training data (¢), estimated 
energy density (shaded), and 
the contour evolution (white 
curve) in appropriate 2D 
projections. The evolving 
contour — see Fig. 8 — is 
constrained to the domains of 
low energy induced by the 


training data | | 
Projection ontoist and 2nd Projection onto 2nd and 4th 
principal component principal component 
4 Statistical Priors for Level Set Representations 


Parametric representations of shape as those presented above have numerous 
favorable properties; in particular, they allow to represent rather complex shapes 
with a few parameters, resulting in low memory requirements and low computation 
time. Nevertheless, the explicit representation of shape has several drawbacks: 


e The representation of explicit shapes typically depends on a specific choice 
of representation. To factor out this dependency in the representation and 
in respective algorithms gives rise to computationally challenging problems. 
Determining point correspondences, for example, becomes particularly difficult 
for shapes in higher dimensions (e.g., surfaces in 3D). 

¢ In particular, the evolution of explicit shape representations requires sophisti- 
cated numerical regridding procedures to assure an equidistant spacing of control 
points and prevent control point overlap. 

¢ Parametric representations are difficult to adapt to varying topology of the 
represented shape. Numerically, topology changes require sophisticated splitting 
and remerging procedures. 

¢« A number of recent publications [11, 49,59] indicate that in contrast to explicit 
shape representations, the implicit representation of shape allows to compute 
globally optimal solutions to shape inference for large classes of commonly used 
energy functionals. 


A mathematical representation of shape which is independent of parameteriza- 
tion was pioneered in the analysis of random shapes by Fréchet [45] and in the 
school of mathematical morphology founded by Matheron and Serra [64, 94]. The 
level set method [39,72] provides a means of propagating contours C (independent 
of parameterization) by evolving associated embedding functions @ via partial 
differential equations — see Fig. 10 for a visualization of the level set function 
associated with a human silhouette. It has been adapted to segment images based 
on numerous low-level criteria such as edge consistency [10, 56, 63], intensity 
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Fig. 10 The level set method 
is based on representing : 
shapes implicitly as the zero Be ah. af 
level set of a eS 
higher-dimensional 
embedding function 


homogeneity [12, 101], texture information [9, 51,73, 81], and motion information 
[33]. 

In this section, we will give a brief insight into shape modeling and shape priors 
for implicit level set representations. Parts of the following text were adopted from 
[34, 35, 82]. 


Shape Distances for Level Sets 


The first step in deriving a shape prior is to define a distance or dissimilarity measure 
for two shapes encoded by the level set functions ¢; and ¢. We shall briefly discuss 
three solutions to this problem. In order to guarantee a unique correspondence 
between a given shape and its embedding function ¢, we will in the following 
assume that @ is a signed distance function, i.e., ¢ > 0 inside the shape, d < 0 
outside, and |V¢| = 1 almost everywhere. A method to project a given embedding 
function onto the space of signed distance functions was introduced in [98]. 

Given two shapes encoded by their signed distance functions ¢, and ¢2, a simple 
measure of their dissimilarity is given by their L2-distance in Q2 [62]: 


i ($1 — $2)? dx. (17) 
Q 


This measure has the drawback that it depends on the domain of integration (2. The 
shape dissimilarity will generally grow if the image domain is increased — even if 
the relative position of the two shapes remains the same. Various remedies to this 
problem have been proposed. We refer to [32] for a detailed discussion. 

An alternative dissimilarity measure between two implicitly represented shapes 
represented by the embedding functions ¢, and @2 is given by the area of the 
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symmetric difference [14, 15,77]: 


Ce em / (Hoi (x) — Hoo(x))? ax. (18) 
Q 


In the present work, we will define the distance between two shapes based on this 
measure, because it has several favorable properties. Beyond being independent 
of the image size (2, measure (18) defines a distance on the set of shapes: it is 
nonnegative, symmetric, and fulfills the triangle inequality. Moreover, it is more 
consistent with the philosophy of the level set method in that it only depends on 
the sign of the embedding function. In practice, this means that one does not need 
to constrain the two level set functions to the space of signed distance functions. It 
can be shown [15] that L° and W! norms on the signed distance functions induce 
equivalent topologies as the metric (18). 

Since the distance (18) is not differentiable, we will in practice consider an 
approximation of the Heaviside function H by a smooth (differentiable) version H,. 
Moreover, we will only consider gradients of energies with respect to the L, norm 
on the level set function, because they are easy to compute and because variations 
in the signed distance function correspond to respective variations of the implicitly 
represented curve. In general, however, these do not coincide with the so-called 
shape gradients — see [46] for a recent work on this topic. 


Invariance by Intrinsic Alignment 


One can make use of the shape distance (18) in a segmentation process by adding it 
as a shape prior Eshape(P) = d7(, bo) in a weighted sum to the data term, which 
we will assume to be the two-phase version of (3) introduced in [13]: 


eyes | (I — uy )Ho(a)dx + i (I — u_)? (1— H6(x)) dx + vf |VH¢lax, 
Q Q 


(19) 
Minimizing the total energy 


Evota(@) = Eaata(?) +a Eshape(P) > Eaata(d) +a d*(, go), (20) 


with a weight a > O, induces an additional driving term which aims at maximizing 
the similarity of the evolving shape with a given template shape encoded by the 
function ¢o. 

By construction this shape prior is not invariant with respect to certain transfor- 
mations such as translation, rotation, and scaling of the shape represented by ¢. 

A common approach to introduce invariance (cf. [17, 35, 83]) is to enhance the 
prior by a set of explicit parameters to account for translation by jz, rotation by an 
angle 0, and scaling by o of the shape: 
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This approach to estimate the appropriate transformation parameters has several 
drawbacks: 


¢ Optimization of the shape energy (21) is done by local gradient descent. In 
particular, this implies that one needs to determine an appropriate time step 
for each parameter, chosen so as to guarantee stability of resulting evolution. 
In numerical experiments, we found that balancing these parameters requires a 
careful tuning process. 

¢ The optimization of jz, 0, 0, and ¢ is done simultaneously. In practice, however, 
it is unclear how to alternate between the updates of the respective parameters. 
How often should one iterate one or the other gradient descent equation? In 
experiments, we found that the final solution depends on the selected scheme 
of optimization. 

¢ The optimal values for the transformation parameters will depend on the 
embedding function @. An accurate shape gradient should therefore take into 
account this dependency. In other words, the gradient of (21) with respect to @ 
should take into account how the optimal transformation parameters j1(#), o(@), 
and 0(@) vary with @. 


Inspired by the normalization for explicit representations introducing in (10), 
we can eliminate these difficulties associated with the local optimization of explicit 
transformation parameters by introducing an intrinsic registration process. We will 
detail this for the cases of translation and scaling. Extensions to rotation and other 
transformations are conceivable but will not be pursued here. 


Translation Invariance by Intrinsic Alignment 
Assume that the template shape represented by @¢o is aligned with respect to the 
shape’s centroid. Then we define a shape energy by 


Estape($) = €2(6, $0) = / (HOG Hn) =HawOY de (22) 


Q 


where the function ¢ is evaluated in coordinates relative to its center of gravity 1g 
given by 


H¢ 


———_, 2 
Jo Ho dx au 


Ly = [ sto ax, with h¢ = 


This intrinsic alignment guarantees that the distance (22) is invariant to the location 
of the shape @. In contrast to the shape energy (21), we no longer need to iteratively 
update an estimate of the location of the object of interest. 
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Translation and Scale Invariance via Alignment 

Given a template shape (represented by $9) which is normalized with respect to 
translation and scaling, one can extend the above approach to a shape energy which 
is invariant to translation and scaling: 


Bit =0G, 60 = | (Hé@sx+ tig) = Heo’ dx, (24) 


Q 


where the level set function @ is evaluated in coordinates relative to its center of 
gravity j1y and in units given by its intrinsic scale og defined as 


Nie 


oe ae 
Jog Hodx 


In the following, we will show that functional (24) is invariant with respect to 
translation and scaling of the shape represented by @. Let ¢ be a level set function 
representing a shape which is centered and normalized such that wg = 0 and og = 
1. Let d be an (arbitrary) level set function encoding the same shape after scaling 
by o € Rand shifting by w € R?: 


05 = (fi — py) ho dx) , Where hd = (25) 


Ho(x) = Hd _ 


Indeed, center and intrinsic scale of the transformed shape are given by 
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The shape energy (21) evaluated for p is given by 


Ewpel) = f (HB (05+ 5) — Hoole) dx 
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Therefore, the above shape dissimilarity measure is invariant with respect to 
translation and scaling. 

Note, however, that while this analytical solution guarantees an invariant shape 
distance, the transformation parameters j1g and og are not necessarily the ones 
which minimize the shape distance (21). Extensions of this approach to a larger 
class of invariance are conceivable. For example, one could generate invariance with 
respect to rotation by rotational alignment with respect to the (oriented) principal 
axis of the shape encoded by @. We will not pursue this here. 


Kernel Density Estimation in the Level Set Domain 


In the previous sections, we have introduced a translation and scale invariant shape 
energy and demonstrated its effect on the reconstruction of a corrupted version 
of a single familiar silhouette the pose of which was unknown. In many practical 
problems, however, we do not have the exact silhouette of the object of interest. 
There may be several reasons for this: 


¢ The object of interest may be three-dimensional. Rather than trying to reconstruct 
the three-dimensional object (which generally requires multiple images and the 
estimation of correspondence), one may learn the two-dimensional appearance 
from a set of sample views. A meaningful shape dissimilarity measure should 
then measure the dissimilarity with respect to this set of projections — see the 
example in Fig. 8. 

¢ The object of interest may be one object out of a class of similar objects (the 
class of cars or the class of tree leafs). Given a limited number of training shapes 
sampled from the class, a useful shape energy should provide the dissimilarity of 
a particular silhouette with respect to this class. 

¢ Evena single object, observed from a single viewpoint, may exhibit strong shape 
deformation — the deformation of a gesticulating hand or the deformation which 
a human silhouette undergoes while walking. In the following, we will assume 
that one can merely generate a set of stills corresponding to various (randomly 
sampled) views of the object of interest for different deformations — see Fig. 11. 
In the following, we will demonstrate that — without constructing a dynamical 
model of the walking process — one can exploit this set of sample views in order 
to improve the segmentation of a walking person. 


In the above cases, the construction of appropriate shape dissimilarity measures 
amounts to a problem of density estimation. In the case of explicitly represented 
boundaries, this has been addressed by modeling the space of familiar shapes 
by linear subspaces (PCA) [22] and the related Gaussian distribution [36], by 
mixture models [21] or nonlinear (multimodal) representations via simple models 
in appropriate feature spaces [27, 28]. 
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Selected sample shapes from a set of a walking silhouettes 


Fig. 11 Density estimated using a kernel density estimator for a projection of 100 silhouettes of 
a walking person (see above) onto the first three principal components 


For level set-based shape representations, it was suggested [62, 84, 100] to fit 
a linear subspace to the sampled signed distance functions. Alternatively, it was 
suggested to represent familiar shapes by the level set function encoding the mean 
shape and a (spatially independent) Gaussian fluctuation at each image location [83]. 
These approaches were shown to capture some shape variability. Yet, they exhibit 
two limitations: Firstly, they rely on the assumption of a Gaussian distribution which 
is not well suited to approximate shape distributions encoding more complex shape 
variation. Secondly, they work under the assumption that shapes are represented by 
signed distance functions. Yet, the space of signed distance functions is not a linear 
space. Therefore, in general, neither the mean nor the linear combination of a set of 
signed distance functions will correspond to a signed distance function. 

In the following, we will propose an alternative approach to generate a statistical 
shape dissimilarity measure for level set based shape representations. It is based 
on classical methods of (so-called nonparametric) kernel density estimation and 
overcomes the above limitations. 

Given a set of training shapes {¢;};=1..v — such as those shown in Fig. 11 — we 
define a probability density on the space of signed distance functions by integrating 
the shape distances (22) or (24) in a Parzen—Rosenblatt kernel density estimator 
[75,79]: 


P< 1 
PO) & 5 Drew (— 355 d(H, H6). (26) 


i=1 
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The kernel density estimator is among the theoretically most studied density 
estimation methods. It was shown (under fairly mild assumptions) to converge to 
the true distribution in the limit of infinite samples (and 0 — 0); the asymptotic 
convergence rate was studied for different choices of kernel functions. 

It should be pointed out that the theory of classical nonparametric density 
estimation was developed for the case of finite-dimensional data. It is beyond the 
scope of this work to develop a general theory of probability distributions and 
density estimation on infinite-dimensional spaces (including issues of integrability 
and measurable sets). For a general formalism to model probability densities on 
infinite-dimensional spaces, we refer the reader to the theory of Gaussian processes 
[76]. In our case, an extension to infinite-dimensional objects such as level set 
surfaces @ : (2 — R could be tackled by considering discrete (finite-dimensional) 
approximations {¢;; € R};=1,...v,j=1,...m Of these surfaces at increasing levels of 
spatial resolution and studying the limit of infinitesimal grid size (i.e., N, M — oo). 
Alternatively, given a finite number of samples, one can apply classical density 
estimation techniques efficiently in the finite-dimensional subspace spanned by the 
training data [82]. 

Similarly respective metrics on the space of curves give rise to different kinds 
of gradient descent flows. Recently researchers have developed rather sophisticated 
metrics to favor smooth transformations or rigid body motions. We refer the reader 
to [16,97] for promising advances in this direction. In the following we will typically 
limit ourselves to L2 gradients. 

There exist extensive studies on how to optimally choose the kernel width o 
based on asymptotic expansions such as the parametric method [37], heuristic esti- 
mates [95, 104], or maximum likelihood optimization by cross validation [18, 42]. 
We refer to [40, 96] for a detailed discussion. For this work, we simply fix o7 to be 
the mean squared nearest-neighbor distance: 


eee. 
= pe min d°(H¢:, 9). (27) 


i=] 


The intuition behind this choice is that the width of the Gaussians is chosen such 
that on the average the next training shape is within one standard deviation. 

Reverting to kernel density estimation resolves the drawbacks of existing 
approaches to shape models for level set segmentation discussed above. In 
particular: 


¢ The silhouettes of a rigid 3D object or a deformable object with few degrees 
of freedom can be expected to form fairly low-dimensional manifolds. The 
kernel density estimator can capture these without imposing the restrictive 
assumption of a Gaussian distribution. Figure 11 shows a 3D approximation of 
our method: We simply projected the embedding functions of 100 silhouettes 
of a walking person onto the first three eigenmodes of the distribution. The 
projected silhouette data and the kernel density estimate computed in the 3D 
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subspace indicate that the underlying distribution is not Gaussian. The estimated 
distribution (indicated by an isosurface) shows a closed loop which stems from 
the fact that the silhouettes were drawn from an essentially periodic process. 

¢ Kernel density estimators were shown to converge to the true distribution in 
the limit of infinite (independent and identically distributed) training samples 
[40, 96]. In the context of shape representations, this implies that our approach is 
capable of accurately representing arbitrarily complex shape deformations. 

e By not imposing a linear subspace, we circumvent the problem that the space 
of shapes (and signed distance functions) is not a linear space. In other words, 
Kernel density estimation allows to estimate distributions on nonlinear (curved) 
manifolds. In the limit of infinite samples and kernel width o going to zero, the 
estimated distribution is more and more constrained to the manifold defined by 
the shapes. 


Gradient Descent Evolution for the Kernel Density Estimator 


In the following, we will detail how the statistical distribution (26) can be used to 
enhance level set based segmentation process. As for the case of parametric curves, 
we formulate level set segmentation as a problem of Bayesian inference, where the 
segmentation is obtained by maximizing the conditional probability: 


PU |) P@) 


P@|D= Pi) 


(28) 


with respect to the level set function ¢, given the input image /. For a given image, 
this is equivalent to minimizing the negative log-likelihood which is given by a sum 
of two energies: 


E(¢) = Edata(d) - Eshape(), (29) 
with 
E snape() = —log P(9). (30) 


Minimizing the energy (29) generates a segmentation process which simulta- 
neously aims at maximizing intensity homogeneity in the separated phases and 
a similarity of the evolving shape with respect to all the training shapes encoded 
through the statistical estimator (26). 

Gradient descent with respect to the embedding function amounts to the evolu- 
tion: 


dp _ 1 OE data OE shape 


ae a do ap GY 
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Fig. 12 Purely intensity-based segmentation. Various frames show the segmentation of a partially 
occluded walking person generated by minimizing the Chan—Vese energy (19). The walking person 
cannot be separated from the occlusion and darker areas of the background such as the person’s 
shadow 


where the knowledge-driven component is given by 
OE shape _ 2 Qj wd (Hd, H¢;) 
Ip 207 Sa; , 


which simply induces a force in direction of each training shape ¢; weighted by the 
factor: 


(32) 


1 
a; = exp (-s3 d*(H¢, 16) , (33) 


which decays exponentially with the distance from the training shape ¢;. 


Nonlinear Shape Priors for Tracking a Walking Person 


In the following, we apply the above shape prior to the segmentation of a partially 
occluded walking person. To this end, a sequence of a walking figure was partially 
occluded by an artificial bar. Subsequently we minimized energy (19), segmenting 
each frame of the sequence using the previous segmentation as initialization. 
Figure 12 shows that this purely image-driven segmentation scheme is not capable 
of separating the object of interest from the occluding bar and similarly shaded 
background regions such as the object’s shadow on the floor. 

In a second experiment, we manually binarized the images corresponding to the 
first half of the original sequence (frames 1 through 42) and aligned them to their 
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Fig. 13 Segmentation with nonparametric invariant shape prior. Segmentation generated by 
minimizing energy (29) combining intensity information with the shape prior (26). For every 
frame in the sequence, the gradient descent equation was iterated (with fixed parameters), using 
the previous segmentation as initialization. The shape prior permits to separate the walking person 
from the occlusion and darker areas of the background such as the shadow. The shapes in the 
second half of the sequence were not part of the training set 


respective center of gravity to obtain a set of training shape — see Fig. 11. Then we 
ran the segmentation process (31) with the shape prior (26). Apart from adding the 
shape prior, we kept the other parameters constant for comparability. 

Figure 13 shows several frames from this knowledge-driven segmentation. A 
comparison to the corresponding frames in Fig. 12 demonstrates several properties: 


¢ The shape prior permits to accurately reconstruct an entire set of fairly different 
shapes. Since the shape prior is defined on the level set function ¢ — rather than 
on the boundary C (cf. [17]) — it can easily handle changing topology. 

e The shape prior is invariant to translation such that the object silhouette can be 
reconstructed in arbitrary locations of the image. 

¢ The statistical nature of the prior allows to also reconstruct silhouettes which 
were not part of the training set — corresponding to the second half of the images 
shown (beyond frame 42). 


5 Dynamical Shape Priors for Implicit Shapes 
Capturing the Temporal Evolution of Shape 


In the above works, statistically learned shape information was shown to cope for 
missing or misleading information in the input images due to noise, clutter, and 
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occlusion. The shape priors were developed to segment objects of familiar shape in 
a given image. Although they can be applied to tracking objects in image sequences, 
they are not well-suited for this task, because they neglect the temporal coherence 
of silhouettes which characterizes many deforming shapes. 

When tracking a deformable object over time, clearly not all shapes are equally 
likely at a given time instance. Regularly sampled images of a walking person, 
for example, exhibit a typical pattern of consecutive silhouettes. Similarly, the 
projections of a rigid 3D object rotating at constant speed are generally not 
independent samples from a statistical shape distribution. Instead, the resulting set 
of silhouettes can be expected to contain strong temporal correlations. 

In the following, we will present temporal statistical shape models for implicitly 
represented shapes that were first introduced in [26]. In particular, the shape 
probability at a given time depends on the shapes observed at previous time 
steps. The integration of such dynamical shape models into the segmentation 
process can be elegantly formulated within a Bayesian framework for level set 
based image sequence segmentation. The resulting optimization by gradient descent 
induces an evolution of the level set function which is driven both by the intensity 
information of the current image as well as by a dynamical shape prior which relies 
on the segmentations obtained on the preceding frames. Experimental evaluation 
demonstrates that the resulting segmentations are not only similar to previously 
learned shapes, but they are also consistent with the temporal correlations estimated 
from sample sequences. The resulting segmentation process can cope with large 
amounts of noise and occlusion because it exploits prior knowledge about temporal 
shape consistency and because it aggregates information from the input images over 
time (rather than treating each image independently). 


Level Set-Based Tracking via Bayesian Inference 


Statistical models can be estimated more reliably if the dimensionality of the 
model and the data are low. We will therefore cast the Bayesian inference in a 
low-dimensional formulation within the subspace spanned by the largest principal 
eigenmodes of a set of sample shapes. We exploit the training sequence in a twofold 
way: Firstly, it serves to define a low-dimensional subspace in which to perform 
estimation. And secondly, within this subspace we use it to learn dynamical models 
for implicit shapes. For static shape priors this concept was already used in [82]. 

Let {¢1,..., ny} be a temporal sequence of training shapes. (We assume that all 
training shapes ¢; are signed distance functions. Yet an arbitrary linear combination 
of eigenmodes will in general not generate a signed distance function. While the 
discussed statistical shape models favor shapes which are close to the training shapes 
(and therefore close to the set of signed distance functions), not all shapes sampled 
in the considered subspace will correspond to signed distance functions.) Let 
denote the mean shape and y,..., W, the n largest eigenmodes withn < N. We 
will then approximate each training shape as 
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Gi (x) = do(x) + Do ony Hj (2), (34) 
j=l 
where 
ai; = (61 — 0, Vj) = J — ho) Wj dx. (35) 


Such PCA-based representations of level set functions have been successfully 
applied for the construction of statistical shape priors in [62, 82, 84, 100]. In the 
following, we will denote the vector of the first n eigenmodes as w = (W1,..., Wn). 
Each sample shape ¢; is therefore approximated by the n-dimensional shape vector 
a; = (Qj1,..., Oj»). Similarly, an arbitrary shape ¢ can be approximated by a shape 
vector of the form 


ay = (b— do, ¥). (36) 


In addition to the deformation parameters a, we introduce transformation 
parameters 0, and we introduce the notation: 


bu,o(X) = $o(Tox) + 0! (Tox), (37) 


to denote the embedding function of a shape generated with deformation parameters 
a and transformed with parameters 0. The transformations Tg can be translation, 
rotation, and scaling (depending on the application). 

With this notation, the goal of image sequence segmentation within this subspace 
can be stated as follows: Given consecutive images J; : Q — R from an 
image sequence and given the segmentations o.,-; and transformations 6. tl 
obtained on the previous images /).,-;, compute the most likely deformation @,and 
transformation 6, by maximizing the conditional probability: 


PU: | at, 6,) P (ar, 6; | O41, Bi-4) 
PC; | G14, O14) 


Pia, 0; | I, Ol1:1-1, Ai:+-1) = (38) 


The key challenge, addressed in the following, is to model the conditional probabil- 
ity: 


P(e, 0, | Oi. Bi-1) (39) 


which constitutes the probability for observing a particular shape aw, and a particular 
transformation 6, at time t, conditioned on the parameter estimates for shape and 
transformation obtained on previous images. 
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Linear Dynamical Models for Implicit Shapes 


For realistic deformable objects, one can expect the deformation parameters a, 
and the transformation parameters 6, to be tightly coupled. Yet, we want to learn 
dynamical shape models which are invariant to the absolute translation, rotation, 
etc. To this end, we can make use of the fact that the transformations form a 
group which implies that the transformation 6, at time ¢ can be obtained from 
the previous transformation 6;-; by applying an incremental transformation AQ;: 
To,x = Tao, To,_,x. Instead of learning models of the absolute transformation @,, 
we can simply learn models of the update transformations A@, (e.g., the changes in 
translation and rotation). By construction, such models are invariant with respect to 
the global pose or location of the modeled shape. 

To jointly model transformation and deformation, we simply obtain for each 
training shape in the learning sequence the deformation parameters o; and the 
transformation changes A6@; and define the extended shape vector: 


a; 
B,:= te (40) 


We will then impose a linear dynamical model of order k to approximate the 
temporal evolution of the extended shape vector: 


An 1 a 
P(B, | By:4) % exp (-3 vy 'y) , (41) 
where 


v=B,—M—AjB,, — 42B,5..-— AcBe4- (42) 


Various methods have been proposed in the literature to estimate the model param- 
eters given by the mean ym and the transition and noise matrices Aj,..., Ax, U. 
We applied a stepwise least squares algorithm proposed in [71]. Using dynamical 
models up to an order of 8, we found that according to Schwarz’s Bayesian criterion 
[92], our training sequences were best approximated by an autoregressive model of 
second order (k = 2). 

Figure 14 shows a sequence of statistically synthesized embedding functions 
and the induced contours given by the zero level line of the respective surfaces — 
for easier visualization, the transformational degrees are neglected. In particular, 
the implicit representation allows to synthesize shapes of varying topology. The 
silhouette on the bottom left of Fig. 14, for example, consists of two contours. 
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Fig. 14 Synthesis of implicit dynamical shapes. Statistically generated embedding surfaces 
obtained by sampling from a second-order autoregressive model and the contours given by the zero 
level lines of the synthesized surfaces. The implicit representation allows the embedded contour to 
change topology (bottom left image) 


Variational Segmentation with Dynamical Shape Priors 


Given an image /; from an image sequence and given a set of previously segmented 
shapes with shape parameters o).,-; and transformation parameters 6).,—;, the goal 
of tracking is to maximize the conditional probability (38) with respect to shape a; 
and transformation 0,. This can be performed by minimizing its negative logarithm, 
which is — up to a constant — given by an energy of the form 


E(a,, 6,) = Edata(:, 0;) ai E shape (01. 6,). (43) 


For the data term we use the model in (3) with independent intensity variances: 


L = 2 I _ 2 
Eaata(0t, 6) = i (“a +log ai) Ha,0, + (Soo +108 a) 


i 203 
(1—Hdze,.6,) ax. (44) 
Using the autoregressive model (41), the shape energy is given by 


1 
E shape (tr, 0:) = vixoly, (45) 


with v defined in (42). 
The total energy (43) is easily minimized by gradient descent. For details we 
refer to [26]. 
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25% noise 50% noise 75% noise 90% noise 


Fig. 15 Images from a sequence with increasing amount of noise 


Segmentation results for 75% noise 


Segmentation results for 90% noise 


Fig. 16 Variational image sequence segmentation with a dynamical shape prior for various 
amounts of noise. Ninety percent noise means that nine out of ten intensity values were replaced by 
a random intensity from a uniform distribution. The statistically learned dynamical model allows 
for reliable segmentation results despite prominent amounts of noise 


Fig. 17 Tracking in the presence of occlusion. The dynamical shape prior allows to reliably 
segment the walking person despite noise and occlusion 


Figure shows images from a sequence that was degraded by increasing 
amounts of noise. 

Figure 16 shows segmentation results obtained by minimizing (43) as presented 
above. Despite prominent amounts of noise, the segmentation process provides 
reliable segmentations where human observers fail. 

Figure 17 shows the segmentation of an image sequence showing a walking 
person that was corrupted by noise and an occlusion which completely covers 
the walking person for several frames. The dynamical shape prior allows for 
reliable segmentations despite noise and occlusion. For more details and quantitative 
evaluations, we refer to [26]. 
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6 Parametric Representations Revisited: Combinatorial 
Solutions for Segmentation with Shape Priors 


In previous sections we saw that shape priors allow to improve the segmentation and 
tracking of familiar deformable objects, biasing the segmentation process to favor 
familiar shapes or even familiar shape evolution. Unfortunately, these approaches 
are based on locally minimizing respective energies via gradient descent. Since 
these energies are generally non-convex, respective solutions are bound to be locally 
optimal only. As a consequence, they depend on an initialization and are likely to 
be suboptimal in practice. One exception based on implicit shape representations as 
binary indicator functions and convex relaxation techniques was proposed in [31]. 
Yet, the linear interpolation of shapes represented by binary indicator functions does 
not give rise to plausible intermediate shapes such that respective algorithms require 
a large number of training shapes. 

Moreover, while implicit representations like the level set method circumvent the 
problem of computing correspondences between points on either of two shapes, it 
is well-known that the aspect of point correspondences plays a vital role in human 
notions of shape similarity. For matching planar shapes there is abundant literature 
on how to solve the arising correspondence problem in polynomial time using 
dynamic programming techniques [48, 85, 93]. 

Similar concepts of dynamic programming can be employed to localize deformed 
template curves in images. Coughlan et al. [23] detected open boundaries by 
shortest-path algorithms in higher-dimensional graphs. And Felzenszwalb et al. used 
dynamic programming in chordal graphs to localize shapes, albeit not on a pixel 
level. 

Polynomial-time solutions for localizing deformable closed template curves in 
images using minimum ratio cycles or shortest circular paths were proposed in [89], 
with a further generalization presented in [88]. There the problem of determining 
a segmentation of an image J : Q — R that is elastically similar to an observed 
template cc : S' > R? by computing minimum ratio cycles 


r:si+oaxs! (46) 


in the product space spanned by the image domain Q and template domain S!. See 
Fig. 18 for a schematic visualization. All points along this circular path provide a 
pair of corresponding template point and image pixel. In this manner, the matching 
of template points to image pixels is equivalent to the estimation of orientation- 
preserving cyclic paths, which can be solved in polynomial time using dynamic 
programming techniques such as ratio cycles [86] or shortest circular paths [91]. 

Figure 19 shows an example result obtained with this approach: The algorithm 
determines a deformed version (right) of a template curve (left) in an image (center) 
in globally optimal manner. An initialization is no longer required and the best 
conceivable solution is determined in polynomial time. 
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Fig. 18 A polynomial-time solution for matching shapes to images: matching a template curve 
C:S! + R? (left) to the image plane 2 C R? is equivalent to computing an orientation-preserving 
cyclic path T : S! + Q x S! (blue curve) in the product space spanned by the image domain and 
the template domain. The latter problem can be solved in polynomial time — see [89] for details 


Template curve Close-up of input image Optimal segmentation 


Fig. 19 Segmentation with a single template: despite significant deformation and translation, the 
initial template curve (red) is accurately matched to the low-contrast input image. The globally 
optimal correspondence between template points and image pixels is computed in polynomial time 
by dynamic programming techniques [89] 


Figure 20 shows further examples of tracking objects: over long sequences of 
hundreds of frames, the objects of interest are tracked reliably — despite low contrast, 
camera shake, bad visibility, and illumination changes. For further details we refer 
to [89]. 


7 Conclusion 


In the previous sections, we have discussed various ways to impose statistical shape 
priors into image segmentation methods. We have made several observations: 
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Frame 1 Frame 10 


Frame 1 Frame 100 Frame 125 Frame 200 


Fig. 20 Tracking of various objects in challenging real-world sequences [89]. Despite bad 
visibility, camera shake, and substantial lighting changes, the polynomial-time algorithm allows 
to reliably track objects over hundreds of frames (Image data taken from [89]) 


¢ By imposing statistically learned shape information, one can generate segmenta- 
tion processes which favor the emergence of familiar shapes — where familiarity 
is based on one or several training shapes. 

¢ Statistical shape information can be elegantly combined with the input image 
data in the framework of Bayesian maximum a posteriori estimation. Maximizing 
the posterior distribution is equivalent to minimizing a sum of two energies 
representing the data term and the shape prior. A further generalization allows 
to impose dynamical shape priors so as to favor familiar deformations of shape 
in image sequence segmentation. 

e While linear Gaussian shape priors are quite popular, the silhouettes of typical 
objects in our environment are generally not Gaussian distributed. In contrast 
to linear Gaussian priors, nonlinear statistical shape priors based on Parzen— 
Rosenblatt kernel density estimators or based on Gaussian distributions in 
appropriate feature spaces [28] allow to encode a large variety of rather distinct 
shapes in a single shape energy. 

e Shapes can be represented explicitly (as points on the object’s boundary or 
surface) or implicitly (as the indicator function of the interior of the object). They 
can be represented in a spatially discrete or a spatially continuous setting. 

¢ The choice of shape representation has important consequences regarding the 
question which optimization algorithms are employed and whether respec- 
tive energies can be minimized locally or globally. Moreover, different shape 
representations give rise to different notions of shape similarity and shape inter- 
polation. As a result, there is no single ideal representation of shape. Ultimately 
one may favor hybrid representations such as the one proposed in [90]. It 
combines explicit and implicit representations allowing cost functions which 
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represent properties of both the object’s interior and its boundary. Subsequent 
LP relaxation provides minimizers of bounded optimality. 


Cross-References 
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Abstract 


Motions of physical objects relative to a camera as observer naturally occur in 
everyday lives and in many scientific applications. Optical flow represents the 
corresponding motion induced on the image plane. This paper describes the basic 
problems and concepts related to optical flow estimation together with mathe- 
matical models and computational approaches to solve them. Emphasis is placed 
on common and different modeling aspects and to relevant research directions 
from a broader perspective. The state of the art and corresponding deficiencies 
are reported along with directions of future research. The presentation aims at 
providing an accessible guide for practitioners as well as stimulating research 
work in relevant fields of mathematics and computer vision. 


1 Introduction 
Motivation, Overview 


Motion of image data belongs to the crucial features that enable low-level image 
analysis in natural vision systems and in machine vision systems and the analysis 
of a major part of stored image data in the format of videos, as documented, 
for instance, by the fast increasing download rate of YouTube. Accordingly, 
image motion analysis has played a key role from the beginning of research in 
mathematical and computational approaches to image analysis. 

Figure | illustrates few application areas of image processing, among many 
others, where image motion analysis is deeply involved. Mathematical models for 
analyzing such image sequences boil down to models of a specific instance of the 
general data analysis task, that is, to fuse prior knowledge with information given 
by observed image data. While adequate prior knowledge essentially depends on the 
application area as Fig. 1 indicates, the processing of observed data mainly involves 
basic principles that apply to any image sequence. Correspondingly, the notion of 
optical flow, informally defined as determining the apparent instantaneous velocity 
of image structure, emphasizes the application-independent aspects of this basic 
image analysis task. 

Due to this independency, optical flow algorithms provide a key component for 
numerous approaches to applications across different fields. Major examples include 
motion compensation for video compression, structure from motion to estimate 3-D 
scene layouts from image sequences, visual odometry, and incremental construction 
of mappings of the environment by autonomous systems, estimating vascular wall 
shear stress from blood flow image sequences for biomedical diagnosis, to name 
just a few. 

This chapter aims at providing a concise and up-to-date account of mathematical 
models of optical flow estimation. Basic principles are presented along with various 
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Fig. 1 Some application areas of image processing that essentially rely on image motion analysis. 
Left: scene analysis (depth, independently moving objects) with a camera mounted in a car. Center: 
flow analysis in remote sensing. Right: measuring turbulent flows by particle image velocimetry 


prior models. Application-specific aspects are only taken into account at a general 
level of mathematical modeling (e.g., geometric or physical prior knowledge). 
Model properties favoring a particular direction of modeling are highlighted, while 
keeping an eye on common aspects and open problems. Conforming to the editor’s 
guidelines, references to the literature are confined to a — subjectively defined — 
essential minimum. 


Organization 


Section 2 introduces a dichotomy of models used to present both essential dif- 
ferences and common aspects. These classes of models are presented in Sects. 3 
and 4. The former class comprises those algorithms that perform best on current 
benchmark datasets. The latter class becomes increasingly more important in 
connection with motion analysis of novel, challenging classes of image sequences 
and videos. While both classes merely provide different viewpoints on the same 
subject — optical flow estimation and image motion analysis — distinguishing them 
facilitates the presentation of various facets of relevant mathematical models in 
current research. Further relationships, unifying aspects together with some major 
open problems and research directions, are addressed in Sect. 5. 


2 Basic Aspects 
Invariance, Correspondence Problem 


Image motion computation amounts to define some notion of invariance and the 
recognition in subsequent time frames of corresponding objects, defined by local 
prominent image structure in terms of a feature mapping g(x) whose values are 
assumed to be conserved during motion. As Fig. 2, left panel, illustrates, invariance 
only holds approximately due to the imaging process and changes of viewpoint and 
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{g(x D}ichn) {g(x/,t+6t)} je[n) 


Fig. 2 Left: image motion can only be computed by recognizing objects as the same in 
subsequent time frames, based on some notion of equivalence (invariance) and some distance 
function. In low-level vision, “object” means some prominent local image structure in terms a 
feature mapping g(x,t) € R’, p > 1. The correspondence problem amounts to compute a 
corresponding assignment {g(x', t)}ietmy > {g(x/,t + 6r)} j€[n]- The corresponding objective 
defines the data term of a variational approach. Right: differential approaches to image motion 
computation are based on smooth feature mappings g(x,f) and aim at solving the assignment 
problem g(x(t),t) > g(x(t + 5t),t + 51). The figure illustrates the basic case of a scalar- 
valued signal g(x,f) translating with constant speed u and the estimate (14) based on the 
differential motion approach, as discussed in section “Assignment Approach, Differential Motion 
Approach” 


illumination. Consequently, some distance function 
p(g(x!,t + 8t) — g(x',)) (1) 


has to be used in order to compute an optimal assignment 


{e(x'!, t)sie(m| > {g(x/,t + 5t)} je[n}- (2) 


A vast literature exists on definitions of feature mappings g(x, f), distance functions, 
and their empirical evaluation in connection with image motion. Possible definitions 
include 


¢ Image gray value or color, 

¢ Gray value or color gradient, 

¢ Output of analytic band-pass filters (e.g., [1,2]), 

¢ More complex feature descriptors including SIFT [3] and SURF [4], 
¢ Censor voting, [5], local patches or feature groupings, 


together with a corresponding invariance assumption, i.e., that g(x, 1) is conserved 
during motion (cf. Fig. 2, left panel). Figure 3 illustrates the most basic approaches 
used in the literature. Recent examples adopting a more geometric viewpoint on 
feature descriptors and studying statistical principles of patch similarity include 
[6,7]. 

For further reference, some basic distance functions p: R? > R4+ are introduced 
below that are commonly applied in connection with feature mappings g(x) and 
partly parametrized by A > 0 and 0 < e < 1. For closely related functions and the 
nomenclature in computer vision, see, e.g., [8]. 
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Fig. 3 (a) Lab scene (©cMuU image database) and (b) gradient magnitude that provides the basis for a 
range of feature mappings g(x, ¢). The image section indicated in (a) is shown in (c), and (d) shows 
the same section extracted from (b). Panels (e) and (f) illustrate these sections as surface plots. 
Panel (g) shows a feature map responding to crossing gray-value edges. (c), (d), and (g) correspond 
to the most basic examples of feature mappings g(x, tf) used in the literature to compute image 
motion, based on a corresponding invariance assumption (cf. Fig. 2, left panel) that is plausible for 
video frame rates 


p3(z) := lzll? squared £7 distance, (3a) 
P2(z) := |lz|l £? distance, (3b) 
2,¢(Z) = Vllzil2 + e2 —€é smoothed £7 distance, (3c) 
pi(z) = [Iza £! distance, (3d) 
Pie(Z) = = P2,¢(Zi) smoothed £! distance, (3e) 
ielp] 

p2,4(Z) = min{||z||7, 7} truncated squared £7 distance, (3f) 
2,1,e(Z) -= —e log jens + a) smoothed tr. sq. ¢? distance. (3g) 


Figure 4 illustrates these convex and non-convex distance functions. Functions 
Pi. and 2, constitute specific instances of the general smoothing principle to 
replace a lower semicontinuous, positively homogeneous, and sublinear function 
p(z) by a smooth proper convex function p;(z), with Ho eeelas €) = p(z) (cf., 

é 


e.g., [9]). Function p24 -(z) utilizes the log-exponential function [10, Ex. 1.30] to 
uniformly approximate 2, ase \, 0. 
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Fig. 4 Left: convex distance functions (3a)—-(3c). Center: level lines of the distance function 
P\.¢ Be). Right: non-convex distance functions (3f) and (3g) 


Assignment Approach, Differential Motion Approach 


Definitions 
Two basic approaches to image motion computation can be distinguished. 


Assignment Approach, Assignment Field This approach aims to determine an 
assignment of finite sets of spatially discrete features in subsequent frames of a 
given image sequence (Fig. 2, left panel). The vector field 


u(x,t), xi =x! +u(x',2t), (4) 


representing the assignment in Eq. (2), is called assignment field. This approach 
conforms to the basic fact that image sequences f(x,t), (x,t) € 2 x[0, T] are 
recorded by sampling frames 


(fk 8t) jen (5) 


along the time axis. 

Assignment approaches to image motion will be considered in Sect. 4. 
Differential Motion Approach, Optical Flow Starting point of this approach is 

the invariance assumption (section “Invariance, Correspondence Problem’) that 

observed values of some feature map g(x, t) are conserved during motion, 


d 
We e.t) = 0. (6) 


Evaluating this condition yields information about the trajectory x(t) that rep- 
resents the motion path of a particular feature value g(x (t)). The corresponding 
vector field 


x(t) = “x0, xER (7) 
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is called motion field whose geometric origin will be described in section “Two- 
View Geometry, Assignment and Motion Fields.” Estimates 


u(x,t) x(t), xe (8) 


of the motion field based on some observed time-dependent feature map g(x, ft) 
are called optical flow fields. 
Differential motion approaches will be considered in Sect. 3. 


Common Aspects and Differences 
The assignment approach and the differential approach to image motion are closely 
related. In fact, for small temporal sampling intervals, 


0<dst <1, (9) 


one may expect that the optical flow field multiplied by df, u(x,t)-6t, closely 
approximates the corresponding assignment field. The same symbol u is therefore 
used in (4) and (8) to denote the respective vector fields. 

A conceptual difference between both approaches is that the ansatz (6) entails 
the assumption of a spatially differentiable feature mapping g(x,t), whereas the 
assignment approach requires prior decisions done at a preprocessing stage that 
localize the feature sets (2) to be assigned. The need for additional processing in the 
latter case contrasts with the limited applicability of the differential approach: The 
highest spatial frequency limits the speed of image motion ||u|| that can be estimated 
reliably: 


a 4 
max {||xlloo, ||u(x)|| lox||:@x € suppg(w), x € 2} < e (10) 


The subsequent section details this bound in the most simple setting for a specific 
but common filter choice for estimating partial derivatives 0; g. 


Differential Motion Estimation: Case Study (1D) 


Consider a scalar signal g(x,t) = f(x, t) moving at constant speed (cf. Fig. 2, right 
panel), 


x(t) =x =u, g(x(t),t) = g(x() + ut, 1). (11) 


Note that the two-dimensional function g(x, tf) is a very special one generated by 
motion. Using the shorthands 


x:=x(0), go(x):= g(x,0), (12) 


g(x, t) corresponds to the translated one-dimensional signal 
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g(x,t) = go(x — ut) (13) 


due to the assumption g(x(¢),t) = g(x(0),0) = go(x). 
Evaluating (6) att = 0, x = x(O) yields 


Pree ae en (14) 
0x£0(X) 


Application and validity of this equation in practice depends on two further aspects: 
Only sampled values of g(x, t) are given, and the right-hand side has to be computed 
numerically. Both aspects are discussed next in turn. 


1. In practice, samples are observed 


(g(k 6x, t6t)}eren = (8k. Otkrew, = x = ot = 1, (15) 


with the sampling interval scaled to | without loss of generality. The Nyquist- 
Shannon sampling theorem imposes the constraint 


supp|Z(@)| C [0,7)°,  @ = (x, a)" (16) 


where 
Go) = Fe) = | (x, te t-G)) dxdt (17) 
R2 


denotes the Fourier transform of g(x, ¢). Trusting in the sensor, it may be savely 
assumed that supp|£o(@,)| C [0,2). But what about the second coordinate t 
generated by motion? Does it obey (16) such that the observed samples (15) 
truly represent the one-dimensional video signal g(x, t)? 

To answer this question, consider the specific case go(x) = sin(@,xX), @, € 
[0, x] — see Fig.5. Equation (13) yields g(x,t) = sin (wx (x — ut)). Condi- 
tion (15) then requires that, for every location x, the one-dimensional time signal 
gx(t) := g(x,f) satisfies supp|Z,(@,)| C [0, 2). Applying this to the example 
yields 


gx(t) = sin(a@;,t + go), OM; = —OxU, Po := OxX, (18) 
and hence the condition 


lo|e[0,7) lul <=. (19) 


x 


It implies that Eq. (14) is only valid if, depending on the spatial frequency @,, 
the velocity u is sufficiently small. 
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Fig. 5 A sinusoid go(x) with angular frequency w, = z/12, translating with velocity u = 2, 
generates the function g(x,t). The angular frequency of the signal g,(t) observed at a fixed 


position x equals |w,| = u-@, = 2/6 due to (18). It meets the upper bound further discussed 
in connection with Fig. 6 that enables accurate numerical computation of the partial derivatives of 


g(x,t) 


This reasoning and the conclusion apply to general functions g(x,t), x € R? 
in the form of (10), which additionally takes into account the effect of derivative 
estimation, discussed next. 

2. Condition (19) has to be further restricted in practice, depending on how the 
partial derivatives of the r.h.s. of Eq. (14) are numerically computed using the 
observed samples (15). The Fourier transform 


F (8% g)(w) = ilo” (0), @ € R?* (20) 


generally shows that taking partial derivatives of order |a| of g(x,t), x € R%, 
corresponds to high-pass filtering that amplifies noise. If g(x, t) is vector valued, 
then the present discussion applies to the computation of partial derivatives 0° g; 
of any component g; (x,t), i € [p]. 

To limit the influence of noise, partial derivatives of the low-pass filtered 
feature mapping g are computed. This removes noise and smoothes the signal, 
and subsequent computation of partial derivatives becomes more accurate. 
Writing g(x), x € R¢*!, instead of g(x,t), x € R%, to simplify the following 
formulas, low-pass filtering of g with the impulse response A(x) means the 
convolution 


els) = x)= fhe Yedy, — Ew) = lo) &) 
: Qu) 
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whose Fourier transform corresponds to the multiplication of the respective 
Fourier transforms. Applying (20) yields 


F (8% gn)(o) = i (h(@) &(@)) = (i!!o%h(w)) &(o). (22) 


Thus, computing the partial derivative of the filtered function g;, can be computed 
by convolving g with the partial derivative of the impulse response 0%. As a 
result, the approximation of the partial derivative of g reads 


a” g(x) & 0% gp(x) = ((0%A) * g)(x). (23) 


The most common choice of / is the isotropic Gaussian low-pass filter 


ee. IP) _ 
hg (x) = onanan %P(- oar) = Il hg (xi), o >0 (24) 


that factorizes (called separable filter) and therefore can be implemented effi- 
ciently. The corresponding filters 0°A,(x), |a| => 1 are called Derivative-of- 
Gaussian (DoG)) filters. 

To examine its effect, it suffices to consider any coordinate due to factoriza- 
tion, that is, the one-dimensional case. Figure 6 illustrates that values 0 > 1 
lead to filters that are sufficiently band limited so as to conform to the sampling 
theorem. The price to pay for effective noise suppression however is a more 
restricted range supp|F(g(x,t))| = [0,@xmax], @x,max « m that observed 
image sequence functions have to satisfy, so as to enable accurate computation of 
partial derivatives and in turn accurate motion estimates based on the differential 
approach. Figure 5 further details and illustrates this crucial fact. 


Assignment or Differential Approach? 

For image sequence functions g(x,f) satisfying the assumptions necessary to 
evaluate the key equation (6), the differential motion approach is more conve- 
nient. Accordingly, much work has been devoted to this line of research up to 
now. In particular, sophisticated multiscale representations of g(x,7) enable to 
estimate larger velocities of image motion using smoothed feature mapping g 
(cf. section “Multiscale”). As a consequence, differential approaches rank top at 
corresponding benchmark evaluations conforming to the underlying assumptions 
[11], and efficient implementations are feasible [12, 13]. 

On the other hand, the inherent limitations of the differential approach discussed 
above become increasingly more important in current applications, like optical flow 
computation for traffic scenes taken from a moving car at high speed. Figure 1, right 
panel, shows another challenging scenario where the spectral properties (@,, @;) 
of the image sequence function and the velocity fields to be estimated render 


Optical Flow 1955 


ONIN f------- 


i} 
y 
i} 
visaf 
bls 
y 


Fig. 6 (a) Fourier transform he (w) of the Gaussian low pass (24), 0 = 1. For values o > 1, 


it satisfies the sampling condition supplhe (w)| C [0, 2) sufficiently accurate. (b) The Fourier 
transform of the Derivative-of-Gaussian (DoG) filter “ha (x) illustrates that for |w| < 2/6 (par- 
tial) derivatives are accurately computed, while noise is suppressed at higher angular frequencies. 
(c), (d) The impulse responses g(x, ¢) and 0;h, (x,t) up to size |x|, |t| < 2. Application of the 
latter filter together with 0,,(x, ¢) to the function g(x, t) discussed in connection with Fig. 5 and 
evaluation of Eq. (14) yield the estimate u = 2.02469 at all locations (x,t) where 0,g(x,t) #0 


application of the differential approach difficult, if not impossible. In such cases, 
the assignment approach is the method of choice. 

Combining both approaches in a complementary way seems most promising: 
Robust assignments enable to cope with fast image motions and a differential 
approach turns these estimates into spatially dense vector fields. This point is taken 
up in section “Unifying Aspects: Assignment by Optimal Transport.” 


Basic Difficulties of Motion Estimation 
This section concludes with a list of some basic aspects to be addressed by any 
approach to image motion computation: 


(i) Definition of a feature mapping g assumed to be conserved during motion 
(section “Invariance, Correspondence Problem’’). 

(ii) Coping with lack of invariance of g, change of appearance due to varying 
viewpoint and illumination (sections “Handling Violation of the Constancy 
Assumption” and “Patch Features’). 

(iii) Spatial sparsity of distinctive features (section “Regularization’’). 
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Fig. 7 The basic pinhole 
model of the mathematically 
ideal camera. Scene points X 
are mapped to image points x 
by perspective projection 


(iv) Coping with ambiguity of locally optimal feature matches (section “Assign- 
ment by Displacement Labeling”). 

(v) Occlusion and disocclusion of features. 

(vi) Consistent integration of available prior knowledge, regularization of motion 
field estimation (sections “Geometrical Prior Knowledge” and “Physical Prior 
Knowledge’). 

(vii) Runtime requirements (section “Algorithms”. 


Two-View Geometry, Assignment and Motion Fields 


This section collects few basic relationships related to the Euclidean motion of a 
perspective camera relative to a 3-D scene that induces both the assignment field 
and the motion field on the image plane, as defined in section “Definitions” by (4) 
and (7). Figures 7 and 12 illustrate these relationships. References [14, 15] provide 
comprehensive expositions. 

It is pointed out once more that assignment and motion fields are purely geomet- 
rical concepts. The explicit expressions (43) and (53b) illustrate how discontinuities 
of these fields correspond to discontinuities of depth or to motion boundaries that 
separate regions in the image plane of scene objects (or the background) with 
different motions relative to the observing camera. Estimates of either field will 
be called optical flow, to be discussed in subsequent sections. 


Two-View Geometry 

Scene and corresponding image points are denoted by X € R? and x € R’, 
respectively. Both are incident with the line Ax, A € R, through the origin. Such 
lines are points of the projective plane denoted by y € P*. The components of y 
are called homogeneous coordinates of the image point x, whereas x and X are 
the inhomogeneous coordinates of image and scene points, respectively. Note that y 
stands for any representative point on the ray connecting x and X. In other words, 
when using homogeneous coordinates, scale factors do not matter. This equivalence 
is denoted by 
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/ 


yxy & y=dy’, AF. (25) 


Figure 7 depicts the mathematical model of a pinhole camera with the image plane 
located at X3 = 1. Perspective projection corresponding to this model connects 
homogeneous and inhomogeneous coordinates by 


x= = i) (26) 

x2 3 \y2 
A particular representative y with unknown depth y3 = X3 equals the scene point 
X. This reflects the fact that scale cannot be inferred from a single image. The 3-D 
space IR? \ {0} corresponds to the affine chart {y € P?: y3 4 0} of the manifold P’. 
Similar to representing an image point x by homogeneous coordinates y, it is 
common to represent scene points X € IR? by homogeneous coordinates Y = 


(%, Yo, Y3, Yj" € BP, in order to linearize transformations of 3-D space. The 
connection analogous to (26) is 


1 (© 
X= Cee (27) 
4 Y; 


Rigid (Euclidean) transformations are denoted by {h, R} € SE(3) with translation 
vector fh and proper rotation matrix R € SO(3) characterized by R'R = 
I, detR = +1. Application of the transformation to a scene point X and some 
representative Y reads 


RX+h and OY, 0 := & qe (28) 
whereas the inverse transformation {—R'h, R'} yields 


oi T 
R'(X—h) and Q™'Y, Q'= (ir — "). (29) 


The nonlinear operation (26), entirely rewritten with homogeneous coordinates, 
takes the linear form 


1000 
y=PY, P= ]0100] = U3x3,0), (30) 
0010 


with the projection matrix P and external or motion parameters {h, R}. In practice, 
additional internal parameters characterizing real cameras to the first order of 
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approximation are taken into account in terms of a camera matrix K and the 
corresponding modification of (30), 


y= PY, P = K(J3x3,0). (31) 
As a consequence, the transition to normalized (calibrated) coordinates 
ji=Ky (32) 
corresponds to an affine transformation of the image plane. 
Given an image point x, taken with a camera in the canonical position (30), the 
corresponding ray meets the scene point X; see Figure 12b. This ray projects in a 


second image, taken with a second camera positioned by {h, R} relative to the first 
camera and with projection matrix 


P’ = K’R'(1,—n/), (33) 
to the line /’, on which the projection x’ of X corresponding to x must lie. Turning to 
homogeneous coordinates, an elementary computation shows that the fundamental 
matrix 

F := K''R"fh], K7 (34) 
maps y to the epipolar line |’, 


I! = Fy. (35) 


This relation is symmetrical in that F ' maps y’ to the corresponding epipolar line 
/ in the first image, 


[=F y. (36) 


The epipoles e,e’ are the image points corresponding to the projection centers. 
Because they lie on / and /’ for any x’ and x, respectively, it follows that 


Fe =0, Fle’ =0. (37) 

The incidence relation x’ € J’ algebraically reads (/’, y’) = 0. Hence by (35), 
(y', Fy) =0 (38) 
This key relation constrains the correspondence problem x <> x’ for arbitrary two 


views of the same unknown scene point X. Rewriting (38) in terms of normalized 
coordinates by means of (32) yields 
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(y', Fy) = (K’ ly’, K" FK(K7'y)) = (K’'y’, E(K7'y)) (39) 


with the essential matrix E that, due to (34) and the relation [Kh], ~ K~'[h]xK7!, 
is given by 


E = K'' FK = R'"[h)x. (40) 


Thus, essential matrices are parametrized by transformations {h, R} € SE(3) and 
therefore form a smooth manifold embedded in R?”°. 


Assignment Fields 
Throughout this section, the internal camera parameters K are assumed to be known, 
and hence normalized coordinates (32) are used. As a consequence, 


K=!I (41) 
is set in what follows. 
Suppose some motion h, R of a camera relative to a 3-D scene causes the image 
point x of a fixed scene point X to move to x’ in the image plane. The corresponding 
assignment vector u(x) represents the displacement of x in the image plane, 


x’ =x+u(x), (42) 


which due to (29) and (26) is given by 


1 ie X —h) 1 (xX, 
= ——_—_ > -— ‘ 43 
i als ae e x oH) X3 (% ee) 
Consider the special case of pure translation, i.e., R = J, ri =e',i =1,2,3.Then 
1 X = hy 1 X 
= a 44 
oo X3—h; i ) X3 & oe 


S((i)-!(R)) 3 
= m1) _ ij; . hes ah. 44b 
h,—1 ( (7: Ax X3 oie 


The image point x, where the vector field u vanishes, u(x-) = 0, is called focus of 


expansion (FOE) 
1 i) 
i=—a le i (45) 
hz (i 


Xe corresponds to the epipole y = e since Fe ~ R'[h]xh = 0. 
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Next the transformation is computed of the image plane induced by the motion 
of the camera in terms of projection matrices P = (J,0) and P’ = R'(J,—h) 
relative to a plane in 3-D space 


(n, X)—d =n, X,;+72.X.+n3X3—d =0, (46) 


with unit normal 7, ||”|| = 1 and with signed distance d of the plane from the origin 
0. Setting p = (_", ), Eq. (46) reads 


(p,Y) =0. (47) 


In order to compute the point X on the plane satisfying (46) that projects to the 
image point y, the ray Y(A) = (*7) , A € R, is intersected with the plane. 


(p.YO)) =M€n,y)-d =0 > A=, y= (Oo? =e : I 
(n, y) 1 alts y) 
(48) 
Projecting this point onto the second image plane yields 
u / aT 1 T 1 i 
y=P’YA)=R | y——=(n,y)h) = R' (UI — hn’ )y 
d d (49) 


=: Hy 


Thus, moving a camera relative to a 3-D plane induces a homography (projective 
transformation) H of P* which by virtue of (26) yields an assignment field u(x) 
with rational components. 


Motion Fields 

Motion fields (7) are the instantaneous (differential) version of assignment fields. 
Consider a smooth path {h(t), R(t)} C SE(3) through the identity {0, 7} and the 
corresponding path of a scene point X € R 


X(t) =A(t)+ R()X, X=X(0). (50) 


Let R(t) be given by a rotational axis g € R° and a rotation angle g(t). Using 
Rodrigues’ formula and the skew-symmetric matrix [q]x € s0(3) with @ = g(0) := 
lq ||, matrix R(t) takes the form 


RO) = expttlalx) = 1 + righ, + AO PgR. (51) 
pt (@t) 


Equation (50) then yields 


X(0) =v+[q)xX,  v:=h(0), (52) 
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where v is the translational velocity at t = 0. Differentiating (26) with y = X 
(recall assumption (41)) and inserting (52) give 


d 1 X3X =X xX 1 X — x, X3 
= . y= —[s. : (53a) 
dt X} X3X7 — X1X3 X3 Xo — x2 X3 
HO-Oegemsst) os 
X3 L\y x2 =i + 93X1 + G2X1X2 — Gi x5 


Comparing (53b) to (43) and (44b) shows a similar structure of the translational part 


with FOE 
1 
x= — (") ; (54) 
v3 \v2 


whereas the rotational part merely contributes an incomplete second-order degree 
polynomial to each component of the motion field that does not depend on the scene 
structure in terms of the depth X3. 

Consider the special case of a motion field induced by the relative motion of a 
camera to a 3-D plane given by (46) and write 


1 1 q 
ny xX] 
— = —([n3+ ; ae) 
x3 od ( : ("") (‘') ) a 
Insertion into (53b) shows that the overall expression for the motion fields takes a 
simple polynomial form. 


Early Pioneering Work 


It deems proper to the authors to refer at least briefly to early pioneering work 
related to optical flow estimation, as part of a survey paper. The following references 
constitute just a small sample of the rich literature. 

The information of motion fields, induced by the movement of an observer 
relative to a 3-D scene, was picked out as a central theme more than three 
decades ago [16, 17]. Kanatani [18] studied the representation of SO(3) and 
invariants in connection with the space of motion fields induced by the movement 
relative to a 3-D plane. Approaches to estimating motion fields followed soon, 
by determining optical flow from local image structure [19-24]. Poggio and 
Verri [25] pointed out both the inexpedient, restrictive assumptions making the 
invariance assumption (6) hold in the simple case g(x) = f(x) (e.g., Lam- 
bertian surfaces in the 3-D scene) and the stability of structural (topological) 
properties of motion fields (like, e.g., the FOE (45)). The local detection of 
image translation as orientation in spatiotemporal frequency space, based on the 
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energy and the phase of collections of orientation-selective complex-valued band- 
pass filters (low-pass filters shifted in Fourier space, like, e.g., Gabor filters), was 
addressed by [26-28], partially motivated by related research on natural vision 
systems. 

The variational approach to optical flow was pioneered by Horn and Schunck 
[29], followed by various extensions [30-32] including more mathematically 
oriented accounts [33-35]. The work [36] classified various convex variational 
approaches that have unique minimizers. 

The computation of discontinuous optical flow fields, in terms of piecewise 
parametric representations, was considered by [8, 37], whereas the work [38] 
studied the information contained in correspondences induced by motion 
fields over a longer time period. Shape-based optimal control of flows 
determined on discontinuous domains as control variable was introduced 
in [39], including the application of shape derivative calculus that became 
popular later on in connection with level sets. Markov random fields and the 
Bayesian viewpoint on the nonlocal inference of discontinuous optical flow 
fields were introduced in [40]. The challenging aspects of estimating both 
motion fields and their segmentation in a spatiotemporal framework, together 
with inferring the 3-D structure, have remained a topic of research until 
today. 

This brief account shows that most of the important ideas appeared early 
in the literature. On the other hand, it took many years until first algorithms 
made their way into industrial applications. A lot of work remains to be done 
by addressing various basic and applied research aspects. In comparison to the 
fields of computer vision, computer science, and engineering, not much work 
has been done by the mathematical community on motion-based image sequence 
analysis. 


Benchmarks 


Starting with the first systematic evaluation in 1994 by Baron et al. [41], benchmarks 
for optical flow methods have stimulated and steered the development of new 
algorithms in this field. The Middlebury database [42] further accelerated this trend 
by introducing an online ranking system and defining challenging data sets, which 
specifically address different aspects of flow estimation such as large displacements 
or occlusion. 

The recently introduced KITTI Vision Benchmark Suite [43] concentrates on 
outdoor automotive sequences that are affected by disturbances such as illumination 
changes and reflections, which optical flow approaches are expected to be robust 
against. 

While real imagery requires sophisticated measurement equipment to capture 
reliable reference information, synthetic sequences such as the novel MPI Sintel 
Flow Dataset [44] come with free ground truth. However, enormous efforts are 
necessary to realistically model the scene complexity and effects found in reality. 
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3 The Variational Approach to Optical Flow Estimation 


In contrast to assignment methods, variational approaches to estimating the optical 
flow employ a continuous and dense representation of the variables u : Q +> R?’. 
The model describing the agreement of u with the image data defines the data 
term Ep(u). It is complemented by a regularization term E(u) encoding prior 
knowledge about the spatial smoothness of the flow. Together these terms define 
the energy function E(u), and estimating the optical flow amounts to finding a 
global minimum u, possibly constrained by a set U of admissible flow fields, and 
using an appropriate numerical method: 


inf E(u), E(u) := Ep(u) + Er(u) (56) 


E(u) is non-convex in general, and hence only suboptimal solutions can be 
determined in practice. 

Based on the variational approach published in 1981 by Horn and Schunck [29], 
a vast number of refinements and extensions were proposed in literature. Recent 
comprehensive empirical evaluations [42, 43] reveal that algorithms of this family 
yield best performance. Section “The Approach of Horn and Schunck” introduces 
the approach of Horn and Schunck as reference for the following discussion, 
after deriving the required linearized invariance assumption in section “Differential 
Constraint Equations, Aperture Problem.” 

Data and regularization terms designed to cope with various difficulties in 
real applications are presented in sections “Data Terms” and “Regularization,” 
respectively. Section “Algorithms” gives a short overview over numerical algorithms 
for solving problem (56). Section “Further Extensions” addresses some important 
extensions of the discussed framework. 


Differential Constraint Equations, Aperture Problem 


All variational optical flow approaches impose an invariance assumption on some 
feature vector g(x,t) € IR”, derived from an image sequence f(x, t) as discussed in 
section “Invariance, Correspondence Problem.” Under perfect conditions, any point 
moving along the trajectory x(t) over time ¢ with speed u(x,t) := <x(t) does not 
change its appearance, i.e., 


d 
qe.) = 9. (57) 


Without loss of generality, motion at f = 0 is considered only in what follows. 
Applying the chain rule and dropping the argument tf = 0 for clarity leads to the 
linearized invariance constraint: 
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Je(x)u(x) + 0;g(x) = 0. (58) 


Validity of this approximation is limited to displacements of about | pixel for real 
data as elaborated in section “Common Aspects and Differences,’ which seriously 
limits its applicability. However, section “Multiscale” describes an approach to 
alleviating this restriction, and thus for now it is safe to assume that the assumption 
is fulfilled. 

A least squares solution to (58) is given by (S(x))! (Ig (x)(0;g(x))) where 


S@) = 1) @l,@). (59) 


However, in order to understand the actual information content of equation sys- 
tem (58), the locally varying properties of the Jacobian matrix Jg(x) have to be 
examined: 


rank(J,) = 0: void constraints on u(x) (for g(x, 0) = const.); 
rank(J,) = 1: ill-conditioned constraints, a single component of u(x) is determined only; 
p =rank(J,) = 2: unique solution u(x) = Jy (x)(8:g(x)); 


p > rank(J,) = 2: over-determined and possibly conflicting constraints on u(x), cf. Fig. 8. 


In the case of gray-valued features, g(x) = f(x) € R, (58) is referred to as the 
linearized brightness constancy constraint and imposes only one scalar constraint 
on u(x) € RR’, in the direction of the image gradient Je(x) = (Ve(x))' 40, ie. 


0:8 (x) 


———_———_, 60 
veo ie 


( Vg(x) u(s)) = 
IVe@l 

This limitation which only allows to determine the normal flow component is 
referred to as the aperture problem in the literature. 

Furthermore, for real data, invariance assumptions do not hold exactly, and 
compliance is measured by the data term as discussed in section “Data Terms.” 
Section “Regularization” addresses regularization terms which further incorporate 
regularity priors on the flow so as to correct for data inaccuracies and local 
ambiguities not resolved by (58). 


The Approach of Horn and Schunck 


The approach by Horn and Schunck [29] is described in the following due to its 
importance in the literature, its simple formulation, and the availability of well- 
understood numerical methods for efficiently computing a solution. 
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Fig. 8 Ellipse representation of S = pd g¢ as in (57) for a patch feature vector with p >> 2 (see 
section “Patch Features”). (a) Three synthetic examples with J, having (top to bottom) rank 0, 1, 
and 2, respectively. (b) Real image data with homogeneous (left) and textured (right) region, image 
edges, and corner (middle). (c) Locally varying information content (see section “Differential 
Constraint Equations, Aperture Problem”’) of the path features extracted from (b) 


Model 

Here the original approach [29], expressed using the variational formulation (56), 
is slightly generalized from gray-valued features g(x) = f(x) € R to arbitrary 
feature vectors g(x) € IR”. Deviations from the constancy assumption in (58) are 
measured using a quadratic function pp = pas leading to 


Ep(u) = ; [ pp (Je(x)u(x) + 880) -) dx. (61) 


As for regularization, the quadratic length of the flow gradients is penalized 
using pr = oe to enforce smoothness of the vector field and to overcome 
ambiguities of the data term (e.g., aperture problem; see section “Differential 
Constraint Equations, Aperture Problem’): 


1 
Er(u) = IG? bs pr([lJu(x)|| eax . (62) 


The only parameter o > 0 weights the influence of regularization against the data 
term. 


Discretization 

Finding a minimum of E(u) = Ep(u) + Ep(u) using numerical methods requires 
discretization of variables and data in time and space. To this end, let {x! jie{n] define 
a regular two-dimensional grid in 2, and let g!(x') and g?(x') be the discretized 
versions of g(x,0) and g(x, 1) of the input image sequence, respectively. Motion 
variables u(x’) are defined on the same grid and stacked into a vector u: 


u(x’) = (2o,) i= ) ce R2". (63) 


u(x!) (u2(x!))i etn] 
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The appropriate filter for the discretization of the spatial image gradients 0;g 
strongly depends on the signal and noise properties as discussed in section “Differ- 
ential Motion Estimation: Case Study (1D).” A recent comparison [11] reports that 
a 5-point derivative filter (5{-1, 8, 0, —8, 1}) applied to 5(g! + g’) performs best. 
Temporal gradients are approximated as 0, g(x!) ~ g?(x') — g(x‘). 

As a result, the discretized objective function can be rewritten as 


1 
E(u) = 5||Du + el? + sa ILull’, (64) 


a 
202 


using the linear operators 


Dia Dio cl Lin 
d=( -): o=(:). ua (ie) (65) 
Dpa Dp2 Cp L22 


with data derivatives cj := (0;g;(x'))iefj and Djx := diag ((0¢g; (x'))ie{n). The 
matrix operator L;; applied to variable u approximates the spatial derivative 0, of 
the flow component u; using the 2-tap linear filter {—1, +1} and Neumann boundary 
conditions. 


Solving 

Objective function (64) is strictly convex in uw under mild conditions [33], and 
thus a global minimum of this problem can be determined by finding a solution 
to V, E(u) = 0. This condition explicitly reads 


(D'D+o0~7°L'L)u=—-D'c (66) 


which is a linear equation system of size 2n in u € R2” witha positive definite and 
sparse matrix. A number of well-understood iterative methods exist to efficiently 
solve this class of problems even for large n [45]. 


Examples 

Figure 9 illustrates the method by Horn and Schunck for a small synthetic example. 
The choice of parameter o is a trade-off between smoothing out motion boundaries 
(see Fig. 9b) in the true flow field (Fig. 9a) and sensitivity to noise (Fig. 9d). 


Probabilistic Interpretation 

Considering E(u) as a the log-likelihood function of a probability density function 
gives rise to the maximum a posteriori interpretation of the optimization prob- 
lem (56), 1.e., 


up plulg,o), plulg,o) « exp(—E(u)). (67) 


ue 
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Fig. 9 (a) Synthetic flow field used to deform an image. (b)-(d) Flow field estimated by the 
approach by Horn and Schunck with decreasing strength of the smoothness prior 


As E(u) is quadratic and positive definite due to the assumptions made in 
section “Solving,” this posterior is a Gaussian multivariate distribution 


pulg,.o) = N(u; p, Z) (68) 
with precision (inverse covariance) matrix » — = D'D+o~°L'L and mean 


vector [LL = —Z~'D'¢ that solves (66). 
Examining the conditional distribution of u' € IR? allows to quantify the 


sensitivity of u. To this end a permutation matrix Q = f) e R**" O'O = 1, 
is defined such that u’ = Q;u. Then, fixing Q;u = Q; leads to 


p(w |Qjw) = V'(a', S') (69) 
with ji’ = Q; and 


Si = Q:EQ7 - (0;2Q7) (O27) | (Q:207). (70) 


Using the matrix inversion theorem to invert » block wise according to Q and 
restricting the result to u; reveals 


-1 


Q.5'Q; = (Q E07 -(Q-E07)(Q-Z07) '(@Z07)). aH 
Comparison of (70) to (71) and further analysis yield (for non-boundary pixels) 
$1 =(Q)E1Q)' = (8! +4071) (72) 


with S' = S(x') as defined by (59). Consequently, smaller values of o reduce the 
sensitivity of uv’, but some choice o > 0 is inevitable for singular S’. 
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Data Terms 


Handling Violation of the Constancy Assumption 

The data term as proposed by Horn and Schunck was refined and extended in 
literature in several ways with the aim to cope with the challenging properties 
of image data of real applications; see section “Basic Difficulties of Motion 
Estimation.” 

Changes of the camera viewpoint as well as moving or transforming objects may 
cause previously visible image features to disappear due to occlusion, or vice versa 
to emerge (disocclusion), leading to discontinuous changes of the observed image 
features g(x(t), f) over time and thus to a violation of the invariance constraint (57). 

Surface reflection properties like specular reflections that vary as the viewpoint 
changes, and varying emission or illumination (including shadows), also cause 
appearance to change, in particular in natural and outdoor scenes. 

With some exceptions, most approaches do not explicitly model these cases and 
instead replace the quadratic distance function ps by the convex £?-distance or 
its differentiable approximation (2, to reduce the impact of outliers in regions 
with strong deviation from the invariance assumption. A number of non-convex 
alternatives have been proposed in the literature, including the truncated square 
distance (2,,, which further extend this concept and are often referred to as “robust” 
approaches. 

Another common method is to replace the constancy assumption on the image 
brightness by one of the more complex feature mappings g(x,f) introduced in 
section “Invariance, Correspondence Problem,” or combinations of them. The aim 
is to gain more descriptive features that overcome the ambiguities described in 
section “Differential Constraint Equations, Aperture Problem,” e.g., by including 
color or image structure information from a local neighborhood. Furthermore, 
robustness of the data term can be increased by choosing features invariant to 
specific image transformations. For example, g(x) = V f(x) is immune to additive 
illumination changes. 


Patch Features 
Contrary to the strongly localized brightness feature g(x) = f(x), local image 
patches sampled from a neighborhood N(x) of x, 


8.1) = (F049) ena ER?) P=) (73) 


provide much more reliable information on u in textured image regions. In fact, 
local approaches set Eg(u) = 0 and rely only the information contained in the data 
term. 

The most prominent instance introduced by Lucas and Kanade [19] chooses a 
Gaussian-weighted quadratic distance function, 


: fe |? i i j 
pe, (z) = | ciag(w )42| , WS (WX! — X’))7en (ey (74) 
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and w(x) := exp (— Il «|7/ (207)). Solving the variational problem (56) decomposes 
into n linear systems of dimension 2 each. Furthermore, the sensitivity in terms of 
(72) reduces to 3? = (S‘)! and 


= > w(x! — x4) (JZ (a e(x’)) (75) 


xJ EN (x!) 


equals the so-called structure tensor. At locations with numerically ill- 
conditioned J,, cf. Fig.8 and the discussion in section “Differential Constraint 
Equations, Aperture Problem,’ no flow can be determined reliably which leads 
to possibly sparse results. The works [46, 47] overcome this drawback by 
complementing this data term by a regularization term. 


Multiscale 

As discussed in section “Differential Motion Estimation: Case Study (1D),” the 
range of displacements u(x) that can be accurately estimated is limited to about 
1 pixel which does not conform to the larger magnitude of motion fields typically 
encountered in practical applications. Multiscale methods allow to remove this 
restriction to some extent. They implement a coarse-to-fine strategy for approxi- 
mately determining large displacements on spatially band-limited image data and 
complementing flow details on finer scales. 

The underlying idea is introduced by means of a multiscale representa- 
tion {2 etn] of image data, where / = 0 and/ = n; — | refer to the finest 
and coarsest scale, respectively. More precisely, g!!! is a spatially band-limited 
version of g with @y max < s;a with 1 = so > 8)... S,,-1 > 0. The computation is 
described by the following recursive scheme with u!"](x) = 0: 


© gll(x,t) = hy * g(x + tut), 2) 
* dull := arg min, E(u) on data g!!l(x, r) 
© ul(x) = ul N(x) + bull(x) 


with a suitable approximation of the ideal low-pass filter h; with frequency response 


l@x|loo < S/ 


hi (wx, @) © (76) 


0 otherwise 


Figure 10 demonstrates the method for two simple examples. 

Actual implementations further make use of the band-limited spectrum of the 
filtered data and subsample the data according to the Nyquist-Shannon sampling 
theorem, leading to a data representation referred to as_ resolution pyramid. The 
recursive structure allows in turn to approximate /; by chaining filters with small 
support for computational efficiency. 
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Fig. 10 Multiscale flow estimation: (a) an image (white noise) gl (x, 0) represented at multiscale 
levels 1] = 0 (black), 1 = 3 (blue), and 1 = 6 (red) with s; = 27", i.e., band limited to s;z. (b) 
Estimate ul! (same color encoding as in (a)) of correct constant flow u(x) = 23.3 on multiscale 
level /. (c)-(d) Same as (a)-(b) for a single line of a real image (@Lavision GmbH) as found in particle 
image velocimetry, an optical fluid flow estimation method 


Regularization 


Ill-posed data terms, sensor noise, and other distortions lead to sparse and locally 
inaccurate flow estimates. Variational approaches allow to incorporate priors on the 
motion regularity by means of additional terms Eg(u). For suitable models Ep(u) 
and E(u), accuracy profits from this concept as the global solution to minimization 
problem (56) represents the best flow field according to both observations and 
priors. Furthermore, in contrast to local methods, missing flow information is 
approximately inferred according to the smoothness prior. This is in particular 
essential in connection with ill-posed data terms (cf. section “Differential Constraint 
Equations, Aperture Problem”). 


Regularity Priors 

A number of a priori constraints u € U for flow estimation have been proposed 
in the literature, based on prior knowledge specific to the application domain. 
Examples include 


¢ Inherent geometrical constraints induced by multi-camera setups (section “Geo- 
metrical Prior Knowledge’), 

e Physical properties of flows in experimental fluid mechanics (section “Physical 
Prior Knowledge’). 
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Fig. 11 (a) Synthetic flow field used to deform an image. (b)-(d) Flow field estimated by the 
approach by Horn and Schunck, however with £; — T V-regularization, with decreasing strength 
of the smoothness prior 


Formally, strict compliance with a constraint u € U can be incorporated into the 
variational formulation (56) by means of the corresponding indicator function 


Ex(u) = bu (u). (77) 


In many applications, however, the set U cannot be specified precisely. Then a 
common approach is to replace 6y by a smoother function measuring the distance 
of u to U in some sense, 


Ex(u) = p(u-Tyu). (78) 


For example, the regularization term of the Horn and Schunck approach presented 
in section “The Approach of Horn and Schunck” may be written as 


Eg(u) = ||Lul|? = lu — Trent) (w)llz (79) 


with semi-norm ||x||z := ||Lx|| and set U = ker(L). Generalizations of the 
approach of Horn and Schunck are based on the same L and modify the distance 
function (section “Distance Functions’) or refine them to become locally adaptive 
and anisotropic (section “Adaptive, Anisotropic, and Nonlocal Regularization”). 

Further extensions replace the gradient operator in (62) and its discretization 
L by other operators having a larger space U = ker(L). For example, operators 
involving second-order derivatives Vdiv and Vcurl have been used for flow 
estimation in experimental fluid dynamics [48-50] (cf. section “Physical Prior 
Knowledge’). 


Distance Functions 

Occlusion of objects does not only lead to sudden changes of the projected 
appearance (cf. section “Data Terms”) but also to motion discontinuities whose 
preservation during flow estimation is crucial in many applications and for the 
interpretation of videos. The penalization of large motion gradients J, can be 
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reduced by replacing the quadratic distance function pe in (62) by convex or non- 
convex alternatives; see (3) for some examples. 

Figure 11 demonstrates the effect of replacing the quadratic distance measure of 
the approach by Horn and Schunck (section “The Approach of Horn and Schunck”’) 
by pr = fo. It becomes apparent that motion discontinuities can be better resolved 
than with pr = ps (see Fig. 9). 


Adaptive, Anisotropic, and Nonlocal Regularization 

A further option is to include a priori information on the location and alignment 
of motion discontinuities by using a spatially varying, adaptive, and possibly 
anisotropic norm in (62) 


Ex) =o | pellaCs)Ihvos 8, (80) 
with || Al|w := || AW || and (omitting the dependency on x) 


W = (wie! wre?). (81) 


The normalized orthogonal directions e!,e? € IR? point across and along the 


assumed motion boundary, respectively. The positive eigenvalues w; and w2 control 
relative penalization of flow changes in the according direction. 

A common assumption made in literature, e.g., [36, 51], is that image edges 
and flow discontinuities coincide and facilitate changes of u(x) across the assumed 
boundary e;. For general features g(x), the notion of image edge is here defined 
by choosing e! and e? as the normalized direction e of the largest and smallest 
change of ||J,e||, respectively, given by the eigenvectors of S = Wi g. The 
associated eigenvalues 4; > Az > O of S control the strength of smoothness by 
setting w; = 1—p(A;),i = 1,2 and suitable increasing p(x) € [0, 1] with p(0) = 0. 
This defines an anisotropic and image-driven regularization. Note that for the gray- 
valued case g(x) = f(x) € R, the formulation simplifies to e! = ||Vg||7'Vg, 
A, = ||Vegll\?, and Ay = 0. The class of flow-driven approaches replace the 
dependency on g(x) of the terms above by the flow u(x) to be estimated. This 
nonlinear dependency can be taken into account without compromising convexity 
of the overall variational approach [36]. 

While the approaches so far measure locally the regularity of flows u, approaches 
such as [52] adopt nonlocal functionals for regularization developed in other 
contexts [53-55] for optical flow estimation. Regularization is then more generally 
based on the similarity between all pairs (u(x), u(x’)) with x, x’ € 2, weighted by 
mutual position and feature distances. 
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Further Extensions 


Three extensions of the basic variational approach are sketched: a natural extension 
of spatial regularizers to the spatiotemporal domain (section “Spatiotemporal 
Approach”), regularization based on the two-view geometry (cf. section “Two-View 
Geometry, Assignment and Motion Fields’) and relative rigid motions for computer 
vision applications (section “Geometrical Prior Knowledge’), and a case study of 
PDE-constrained variational optical flow estimation in connection with imaging 
problems in experimental fluid dynamics (section “Physical Prior Knowledge’”’). 


Spatiotemporal Approach 

The preceding discussion reduced the motion estimation problem to determining 
displacements between two image frames only and thus ignored consistencies of 
the flow over time. Although in many applications recording rates are fast compared 
to dynamical changes due to modern sensors, only few approaches exploit this fact 
by introducing temporal smoothness priors. 

The work [35] proposed to process a batch of image frames simultaneously and 
to extend the flow field domain along the time axis u : 2 x [0, T] +> IR?. While data 
terms are independently imposed for each time fr, the smoothness prior is extended 
by a temporal component to 


E(u) := / PR(lSue (x, 0) || w)dxde . (82) 
2x{0,T] 


Here, J,; represents the spatiotemporal derivatives, and pp||-||w is a three- 
dimensional extension of the anisotropic, flow-driven distance function discussed in 
section “Adaptive, Anisotropic, and Nonlocal Regularization.” It allows to account 
for small position changes of moving objects between consecutive frames within 
the support of the regularization term (<1 px) by supporting smoothness along an 
assumed trajectory. 

Larger displacements, however, require matching of temporally associated 
regions, e.g., using a multiscale framework (section “Multiscale”), but then enable 
to regularize smoothness of trajectories over multiple frames as proposed in [56]. 

Online methods are an appealing alternative whenever processing a batch of 
image frames is not feasible due to resource limitations. This approach is addressed 
in section “Probabilistic Modeling and Online Estimation.” 


Geometrical Prior Knowledge 
In applications with a perspective camera as image sensor, the geometrical scene 
structure strongly determines the observed optical flow (section “Two-View Geom- 
etry, Assignment and Motion Fields”). This section briefly addresses the most 
common assumptions made and the constraints that follow. 

Often, a static scene assumption is made, meaning that all visible scene points 
have zero velocity with respect to a world coordinate system. Then the observed 
motion is only induced by the camera moving in the scene. Using the notation 
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epipolar plane 


camera at t = 1 


camera at t = 0 camera at t = 0 camera at t = 1 


optical flow induced by camera motion epipolar constraint 


Fig. 12 (a) Relative motion Q € SE(3) of the camera w.r.t. a world coordinate system causes the 
projection x of a static scene point ¥ = z(x)y to move from x to x’ = x + u(x) in the image 
plane. (b) Any two projections x, x’ of a scene point X are related by the essential matrix E as 
in (84), defining an epipolar plane and their projection, the epipolar lines defined by {x,e} and 
{x’, e’} in the image plane at ¢ = 0 and t = 1, respectively 


introduced in section “Two-View Geometry, Assignment and Motion Fields,” 
the camera motion is denoted by Q € SE(3) (cf. Fig. 12a), parametrized by 
rotation R € SO(3) and translation h € IR°, so that any scene point Y € is 
transported to Y’ ~ Q7'Y. 

The following discussion of common setups and their implications on the 
observed motion implicitly assumes that the scene point is visible in both frames. 
Using assumption (41) for the internal camera parameters allows to work with 
normalized coordinates (32). The point corresponding to x is denoted by x’, due 
to (42). 


Static scene, general motion Let the depth map z(x) : 2 t& R parametrize the 
scene point X := z(x) (7) visible at x in the camera plane in the first frame. 
Then the projected correspondences are given in homogeneous coordinates by 


y ~ POTTY =R' (x) (7) -A), (83) 
see Fig. 12a for an illustration. Figure 13 shows the optical flow field u(x) 
conforming to constraint (83) for a real application. 


It is possible to eliminate the dependency on z(x) that typically is unknown by 
means of the essential matrix E := R" [h]x., leading to the epipolar constraint 


(y’)' Ey =0, (84) 


as illustrated by Fig. 12b. This gives rise to an orthogonal decomposition [58] 
of an observed correspondence x’ into 


Optical Flow 1975 


a b 


— 


Fig. 13 (a) A single frame from an image sequence recorded by a camera moving forward 
through an approximately static scene. (b) Optical flow estimated using the parametrization 
u(x) = u(x; Q, z(x)) according to (83) and global optimization for Q € SE(3), z € R”; see [57] 
for details. Displacement length and direction are encoded by saturation and hue, respectively; see 
color code on the right. (c) Estimated depth parameter z(x) using the color code on the right. Scene 
structure is more evident in this representation, and therefore the spatial smoothness prior on the 
flow was formulated as regularization term on the depth z(x) instead of displacements u(x) 


Ha 4 (85) 


with X/ fulfilling (84) and orthogonal deviations <’ . 
Even without knowing a priori (R,/), Eq.(84) provides a valuable prior: 
Valgaerts et al. [59] propose joint computation of the fundamental matrix F 
related to E by (40) and optical flow constrained via (84). They show that 
estimation of F' is more stable and that flow accuracy is significantly increased. 
Static scene, coplanar camera motion If the camera translates parallel to the image 
plane only, ie., R = J andh = (3) with b € IR?, the observed flow is 
constrained to a locally varying one-dimensional subspace parametrized by the 
inverse depth: 


u(x) =z !(x)b. (86) 


Stereoscopic camera setups fulfill the static scene assumption as they can be 
interpreted as an instantaneous camera motion with baseline ||b||. For details 
see, e.g., [5]. 

Planar and static scene, general camera motion In applications where the scene can 
be (locally) approximated by a plane such that (n, X) — d = 0 for all space 
points X with plane parameters d, n as in (46), all correspondences fulfill 


1 
Jute aa (1 : i") (87) 


where H € IR**? defines a homography — cf. Eq. (49). 


Physical Prior Knowledge 
Imaging of dynamic phenomena in natural sciences encounters often scenarios 
where physical prior knowledge applies. Examples include particle image velocime- 
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try [60] or Schlieren velocimetry [61], where the motion of fluids is observed, 
that is, governed by physical laws. While local methods such as cross-correlation 
methods are commonly used to evaluate the obtained image sequences [60, 62], 
variational approaches [63-65] provide a more appropriate mathematical framework 
for exploiting such prior knowledge and the estimation of physically consistent 
optical flows. 

For instance, the Helmholtz decomposition of vector fields enables to define reg- 
ularizers in terms of higher-order partial flow derivatives in a natural way [48-50]. 
Constraints like incompressibility can be enforced as hard or soft constraints using 
advanced methods of convex programming, to cope with imaging imperfections. 
Conversely, flow field estimates obtained by other image processing methods can 
be denoised so as to restore physically relevant structure [66]. 

A particularly appealing approach exploits directly some equation from fluid 
dynamics that governs the flow as state of the physical system which is observed 
through an imaging sensor [67, 68]. The state is regarded as hidden and only 
observable through the data of an image sequence that depicts the velocity of some 
tracer suspended in the fluid. The variational approach of fitting the time varying 
state to given image sequence data results in a PDE-constrained optimization or 
distributed parameter control problem, respectively. 

As example the approach [67] is sketched based on the Stokes system 


—ywAu+ Vp = fo in Q, (88a) 
divu = 0 in 22, (88b) 
u= fag on 092, (88c) 


that for given fg, fre with f,,(n, fag)ds = 0 (n denotes the outer unit normal 
of the Lipschitz domain £2) has a unique solution u, p under classical assumptions 
[69, Ch. I]. Here fg, fag are not regarded as given data but as control variables, 
to be determined so that the flow uw not only satisfies (88) but fits also given image 
sequence data. To achieve the latter, both the state variables u, p and the control 
variables fo, fag are determined by minimizing in the two-dimensional case d = 2 
the objective 


E(u, p, fax fur) = Evta+ | pi(foax+y | p3((n~, V fag))ds, ay > 0. 


(89) 
The first term Ep(u) denotes a data term of the form (61), and the remaining two 
terms regularize the control variables so as to make the problem well posed. 

For related mathematical issues (e.g., constraint qualification and existence of 
Lagrange multipliers), see [70, Ch. 6] and [71, Ch. 1] and furthermore [70, 72] for 
related work outside the field of mathematical imaging based on the general Navier- 
Stokes system. 
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Algorithms 


The choice of an optimization method for numerically minimizing the func- 
tional (56) depends on the specific formulation of the terms Ep and Ep involved. 
Suitable methods can be broadly classified into 


— Algorithms for minimizing smooth convex functionals, 
— Algorithms for minimizing non-smooth convex functionals, 
— Algorithms for locally minimizing non-convex functionals. 


In view of the typical multiscale implementation of the data term (section “Mul- 
tiscale”) that enables a quadratic approximation at each resolution level, this 
classification is applied to the regularizer Ep only, and each class is discussed 
in turn in the sections to follow. The reader should note that convex non- 
quadratic data terms, as discussed in section “Handling Violation of the 
Constancy Assumption,” can be handled in a similar way as the convex non- 
smooth regularizer below, and a number of closely related alternatives exist 
(e.g., [73]). Since convex programming has been extensively studied in the 
literature, the following presentation is confined to representative case studies 
that illustrate in each case the underlying idea and application of a general 
principle. 


Smooth Convex Functionals 

It is useful to distinguish quadratic and non-quadratic functionals. The approach 
of Horn and Schunck (section “The Approach of Horn and Schunck’”) is a 
basic representative of the former class. Solving the corresponding linear positive 
definite sparse system can be efficiently done by established methods [45]. More 
sophisticated implementations are based on numerical multigrid methods [74]. 
These are optimal in the sense that runtime complexity O(7) linearly depends on 
the problem size n. Dedicated implementations run nearly at video frame rate on 
current PCs. 

For more general data-dependent quadratic regularizers and especially so for 
non-quadratic convex regularizers (cf. section “Adaptive, Anisotropic, and Nonlocal 
Regularization” and [36]), multigrid implementation that achieves such runtimes 
requires some care. See [12, 13, 75] for details and [76] for a general exposi- 
tion. 


Non-smooth Convex Functionals 

This class of optimization problems has received considerable attention in connec- 
tion with mathematical imaging, inverse problems, and machine learning and in 
other fields during the recent years, due to the importance of non-smooth convex 
sparsity enforcing regularization. See [77] for a recent overview. 
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The total variation regularizer 


Eg(u) = TV(u) := sup [ (u, Div v)dx, 
veD 2 


D:= {v € CX(2: R41): ||v()||e <1, Vx € Q}, So 
Div v = (divv!,...,divv“)" 

is a basic representative of the class of non-smooth convex functionals and 

appropriate to expose a general strategy of convex programming that is commonly 

applied: problem splitting into subproblems for which the proximal mapping can be 

efficiently evaluated. 


The simplest anisotropic discretization of (90) that is particularly convenient 
from the viewpoint of convex programming reads 


YS DE ae) - ae), (91) 


ij €E(G) ke[d] 
where {x'}; jn] are the locations indexed vertices V = [n] of a grid graph G = 
(V, E) in 82, and E = E(G) are the corresponding edges connecting adjacent 
vertices resp. locations along the coordinate axes. Defining the vector 
ze RPO! ai; = g(x!) — u(x) (92) 
leads to the reformulation of (91) 


IIzlli. Lu=z (93) 


where the linear system collects all equations of (92). As a consequence, the overall 
discretized problem reads 


min Ep(u) + @||z\|; subjectto Lu—z=0, a>0O (94) 


to which the ADMM approach [78] can be applied that entails a sequence of partial 
minimizations of the augmented Lagrangian corresponding to (94), 


a 
Ly (u,z, w) = Ep(u) + @|lz|]) + (w, Lu —z) + 5 llLu =,(". (95) 


Specifically, with some parameter value A > O and multiplier vector w, the three- 
step iteration 


Ar 
uk*! — arg min Ep(u) + (w*, Lu) + 5llLu —2 |", (96a) 
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7 Ar 
2+! = arg min allz||; — (w*, z) + zie —2\*, (96b) 
wrt! = w* tbe ag tl), (96c) 
is iteratively applied for k = 0,1,2,..., with arbitrary initializations z°, q°, until a 


suitable termination criterion is met [78, Section 3.3.1]. 

Assuming a quadratic form or approximation of Ep(u) at some resolution level 
(section “Multiscale’’), subproblem (96a) amounts to solve a sparse positive definite 
linear system similar to the basic approach of Horn and Schunck, to which a 
multigrid solver can be applied as discussed above. Subproblem (96b) amounts to 
computing the proximal mapping for the £'-norm and hence to perform a simple 
shrinkage operation. See [79, 80] for corresponding surveys. 


Non-convex Functionals 
Similar to the preceding non-smooth convex case, approaches are of interest that 
can be conducted by solving a sequence of simple subproblems efficiently. Clearly, 
convergence to a Jocal minimum can be only expected. In contrast to the simpler 
convex cases above, the absence of parameters is preferable that would have to be set 
properly, to ensure convergence to some local minimum for any initialization. For 
example, Lipschitz constants of gradients are rarely known in practice, and setting 
corresponding parameters savely enough will unduly slow down convergence even 
for smooth problems. 

A general strategy will be outlined next and its application to the non-convex 
extension of the regularizer (91), using the distance function (3f), 


Yo pra (u(x!) — u(x!) (97) 


ij €E(G) 


In order to illustrate graphically the non-convexity of this regularizer from the 
viewpoint of optimization, consider three summands of the “fully” anisotropic 
version of (97), 


YY pra (uel) = ue (x). (98) 


ij €E(G) ke[d] 
defined on edges that meet pairwise in a common vertex, 
2,2 (Uk (x!) — ug (X"2)) + pra (ue (x"2) — Ue (x) + frr,a (ue (x) — Ue (x"4)). (99) 
Setting for simplicity and w.l.o.g. u(x!) = uz (x'4) = 0 to obtain a function of two 
variables ux(x'*), uz (x'3) results in the corresponding graph depicted by Fig. 14. It 


illustrates the presence of many non-strict local minima and that the design of a 
convergent minimization algorithm is not immediate. 
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Fig. 14 Top row, left: two different illustrations of the non-convex, non-smooth objective (99). 
Bottom row, left: the objective (99) smoothed by replacing the distance function po, by p2,,¢ 
with « = 0.2, as defined by (3g). Right panel: sequences of iterates generated by (103) for 30 
random points (z2, 23) ' (initial and final iterates are marked with red and yellow, respectively). 
The regularizer enforces fitting of the components Z),z3 to the data z} = zy = O as well as 
22 = 23. It is robust in the sense that components that are too distant to either of these criteria are 
not affected accordingly 


Next consider a single summand 2,4 (z; — z;) of (98) with two scalar variables 
denoted by z; and z; for simplicity. This function can be decomposed into the 
difference of two proper, lower semicontinuous (Isc), convex functions g and h: 


2,4 (Zi _ z;) = T(Z = zj)" = (rl —z)° = 2,1 (Zi — z))) 
(100) 
= 9(zi,zj) — hi, z;), ped) 
Applying this decomposition to each term of (98) yields 
g(u) — h(u) (101) 


with g(u) = t||Lul|* as in (64), and with h(w) equal to the sum of all edge terms of 
the form h(uj(x!), uz(x/)), ij € E, k € [d], given by (100). 

DC-programming (DC stands for Difference-of-Convex functions [81]) amounts 
to locally minimize (101) by solving a sequence of convex problems, defined by the 
closed affine majorization of the concave part —/A, 


uk tl — arg min g(u) — (h(u‘) ++ iv, u— u*)), v € dn(u'), (102) 
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where dh(u“) denotes the subdifferential of h at u“. This two-step iteration in terms 
of (u«, v*) converges under mild conditions [82]. Smoothing the problem slightly 
by replacing the distance function p2, by p2,,- defined by (3g), and replacing 
accordingly h by h,, yields v¥ = Vh,(uw*) and hence turns (102) into the sequence 
of problems 


uk*! = arg min g(u) — (Vh.(u“),u—u*). (103) 


Taking additionally into account the data term Ep(u) and assuming it (or its 
approximation) has quadratic form at some resolution level (section “Multiscale’’), 
solving (103) amounts to a sequence of Horn and Schunck type problems to which 
numerical multigrid can be applied, due to the simple form g(u) = t||Lul||?. Not 
any single parameter, e.g., for selecting the stepsize, has to be set in order to ensure 
convergence, and available code for a variational method can be directly applied. 
The price to pay for this convenience is a moderate convergence rate. 

Figure [4 illustrates the beneficial effect of smoothing and robustness of the non- 
convex regularizer: Only the components of points (2 ) that are close enough to the 
data z} = ug(x!'!) = zy = u,(x"*) = 0, as specified by A, are fitted to these data. 
For distant points with z2 ~ z3, regularization enforces z2 = z3 or does not affect 
them at all if |z) — z3| is large. 

Applying the scheme (103) to (97) instead of (98) is straightforward. This does 
not affect g(u) but merely V/, in (103), due to replacing the scalar variables in (100) 
by the corresponding vectors. 


4 The Assignment Approach to Optical Flow Estimation 


In this section approaches to determining the assignment field u(x,t) (4) are 
considered that establish the correspondence (2) of a given feature mapping g(x, t) 
in two given images. 

The following sections conform to a classification of these approaches. Both the 
scope and the application areas associated with each class of approaches overlap 
with the variational approach of Sect.3 but otherwise differ. The presentation 
focuses on the former aspects and the essential differences, whereas an in-depth 
discussion of the latter aspects is beyond the scope of this survey. 

Section “Local Approaches” discusses local approaches to the assignment 
problem, whereas the remaining three sections are devoted to global approaches. 
In section “Assignment by Displacement Labeling” the correspondence problem 
is reformulated as a labeling problem so that methods for solving the Maximum 
A Posteriori (MAP) problem with the corresponding Markov random field (MRF) 
model can be applied. Assignment by variational image registration is briefly 
considered in section “Variational Image Registration.” 
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Local Approaches 


Key feature of the class of assignment approaches is the restriction of the set of 
feasible assignment fields u(x) to a finite set. This set is defined by restricting at 
each location {x!}ie[n) € @ the range of u(x') € U(x’) toa finite set U/(x'). 

Local approaches determine the optimal u(x’) independently, i.e., they solve for 
eachi € [n] 


u(x!) € arg min p(g(x', t)g(x' +u,t+ ét)). (104) 
uel (x!) 


The usually small sets |2/(x!)| allow exhaustive search to find an optimal solution. 
Thus, the general distance function p(-,-) is not required to be convex or 
differentiable and allows for more involved formulations. 

Since local methods do not make use of (nonlocal) spatial smoothness priors 
w.r.t. u, they require — and, in fact, solely rely on — discriminative features, typically 
derived from local images patches also used by local variational methods; see (73): 


8D =(FCID) ewer ER? P= ING) (105) 


with some neighborhood N(x‘), e.g., a square region. 

In the following some common choices for p are addressed. For brevity, the 
discussion omits references to x' and some fixed u = u(x!) and puts g! := 
g(x',t!), eg? = g(x! + u,¢t?) with 2? = t! + 8. 

Template-based matching methods compare a template g! pixel wise to a 
potential match g? and derive some similarity measure from it. Direct comparison 
of gray values, 


p(g',g) = p(g! — g*) (106) 


is usually avoided in favor of distance functions which are invariant to brightness or 
geometric changes. Two popular choices are: 


¢ The normalized cross-correlation [83] derives patch features which are invariant 
to global additive and multiplicative changes of g by defining 


=k _ 8 — we") 


k=1,2 107 
a(g*) als 


with mean j(g*) and standard deviation o(g*) of samples {gi } je[p]- Then the 
distance function is defined as 


D napens ae 
pnec(g', 8°) = 1— —(8', 8") = —p3(e' - 8) (108) 
Pp 2p 
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where the last equation follows from (g*, g*) = po?(g*) = p. 
¢ The Census transform creates binary descriptors 


= k _ mt €{0,1?, k=1,2 109 
= (Ys(ej—m)) | (0.13 (109) 


with m* := g(x',t*), which approximate directional derivatives [84] and 
measures the Hamming distance 


pcr(g'.g°) = pi (g' - 8) =(l2' - 2" |l- (110) 


This transformation is in particular invariant to any strictly monotonically 
increasing transformation y: R +> R uniformly applied to all components of g! 
and g?. 


Histogram-based methods relax the pixel-by-pixel comparison in (106) to 
achieve additional invariance to geometric transformations. 

Exemplarily, a method frequently used in medical images registration [85] 
and stereo disparity estimation [86] is detailed. It uses the concept of mutual 
information [87] to measure distances between gray-value probability distributions 
p.(f; g*), k = 1,2, determined as kernel density estimates [88] from the samples 
{ gi } je[p]- Their entropies are given by 


Hie) =— fi Sis) lovi feds k=12 atD 
The joint distribution p;2(f!, £7; g!, g?) is defined accordingly with joint entropy 


H(fi2:g'.g°) = — / Bio(f'. f7:g', 8°) logpia(f'. f7:g'.g°)df'df?. 
(112) 
Then the mutual information defines the distance function 


pui(g', g°) = H(p1; g') + H(po; g”) — Hi2; g'. 8”) (113) 


which shows some robustness against rotation, scaling, and illumination changes. 

Complex approaches such as scale-invariant feature transform (SIFT) [3] and 
speeded-up robust features (SURF) [4] combine several techniques including 
histogram of orientations and multiple resolution to optimize robustness, reliability, 
and speed. 
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Assignment by Displacement Labeling 


Consider again sets U/(x') of assignment vectors as discussed in section “Local 
Approaches.” In contrast to local approaches presented in the previous section, this 
section is devoted to methods that simultaneously select vectors u(x') € U(x’) 
for all locations x‘, i € [n], based on optimization criteria that evaluate desired 
properties of assignment fields u. The feasible set of u is denoted by U := 
UevU(x'). It will be convenient to index locations {x't;cy € 2 by vertices 
i € V = [n] ofa graph G = (V, E£). 

As a consequence of the twofold discretization of both the underlying domain 
2 C R@ and the range of u(x), it makes sense to associate with each location x! an 
integer-valued variable 


£; = £(x') € [mi], m; := |U(x')|, (114) 


whose value determines the assignment vector u(x') € U(x’). This separates the 
problem formulation in terms of the labeling field € := {f;}ievy from the set of 
assignment vectors U/ that may vary, as is further discussed below. 

Analogous to objectives (56) of variational approaches, a functional as criterion 
for labelings £ defines an approach, 


J(L;U) = Jp(€) + Ir) 
=) eG) + Yo GiGi. €;:W), (115) 


ieV ij€E 


together with an algorithm for determining an assignment field wu in terms of a 
minimizing labeling field £. For instance, in view of a data term like (61), a 
reasonable definition of the function g;(-;2/) of (115) is 


gi(li:U) = pp (Je(x' ue, + O:8(x')||-), ue EUR), € € [mi] (116) 


where £; enumerates all possible assignment vectors uy, at x’. However, getting 
back to the differences to the differential approach addressed in section “Common 
Aspects and Differences,” a major motivation of formulation (115) is to disregard 
partial derivatives of the feature map involved in differential variational approaches 
(section “Differential Constraint Equations, Aperture Problem’’) and hence to avoid 
the corresponding limitations discussed in sections “Differential Motion Estimation: 
Case Study (1D)” and “Multiscale.” Rather, data terms Jp are directly defined by 
setting up and evaluating locally possible assignments ug, € U(x') that establish a 
correspondence between local features (2), extracted from the given image pair, 
and by defining costs g;(¢;;U/) accordingly. Notice that no smoothness of 9; is 
required — any distance discussed in section “Local Approaches” may be employed 
as in (116). For a discussion of the distance (113) in this connection, see [89]. 
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The same remarks apply to the definition of Jp in (115). A common choice in 
the literature however is the discrete version of the non-convex regularizer (97) 


ij (E;,€j3U) = frr,, (ue, — ue;3U). (117) 


The reader should notice that the non-convex regularizer (97) has been replaced 
by the combinatorial version (117). Likewise, the non-convex data term (61) has 
been replaced by the discrete-valued term (116). More generally, the problem 
related to the variational approach to cope with the non-convexity of the data term 
(section “Multiscale”) by means of a multiscale implementation (section “Multi- 
scale”), and with the non-convexity of the overall functional by computing a “good” 
local minimum (section “Non-convex Functionals”), has been replaced by the 
combinatorial problem to determine an optimal assignment by minimizing (115). 
This problem is known in the literature as maximum a posteriori (MAP) problem 
w.r.t. the discrete probabilistic graphical model 


pall) = 5 exp(— JUL), Z =) exp(—J(6W)), (118) 
l 


that is the problem to compute the mode arg maxpg(¢;U/) of the Markov random 
field pg defined on the undirected graph G. See [90,91] for background and further 
details. 

Many past and current research activities are devoted to this problem, across 
various fields of computer science and applied mathematics. Approaches range from 
integer programming techniques to various convex relaxations and combinations 
thereof. To get a glimpse of the viewpoint of polyhedral combinatorics on the 
problem to minimize (115), consider a single summand g;(€;;/) and define the 
vector 


gi € R”, 0b, = gi(li;U), L; € [mi], (119) 


whose components specify the finite range of the function g;. Then the problem of 
determining £; corresponding to the minimal value of g;(€;;2/) can be rewritten as 


min (6', y'), (120) 


WEA; 


which is a linear program (LP). Clearly, for general data defining 6’ by (119), the 
vector 7’ minimizing (120) is a vertex of the simplex Ay, Corresponding to the 
indicator vector 4 = (0,...,0,1,0,..., 0)! of the value £;. This reformulation 
can be applied in a straightforward way to the overall problem of minimizing (115), 
resulting in the LP 


ae (121) 
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Fig. 15 Top Frame of a sequence, taken with a fast moving camera from the KITTI benchmark 
(section “Benchmarks’’). Bottom Optical flow estimate based on MAP inference. The disk on the 
right displays the color code of flow vectors. Each image patch localized at x! where a sufficiently 
discriminative feature could be extracted is associated with a set U/(x') of possible assignment 
vectors ug, € U(x'). The displayed assignment field 7 := {uz hiev is determined by a labeling 


field 2 minimizing the functional (115). The latter combinatorial task has been solved to global 
optimality by an approach combining convex relaxation and integer programming [93]. Global 
optimality enables model validation: any deficiencies of the assignment field estimate are solely 
due to the model components, feature extraction, and constraints, as encoded by the MRF (118) 
through J(¢;U/) 


defined over the so-called marginal polytope Mg. This polytope is the convex hull 
of feasible vectors ju, as is the simplex A,,, in (120) for the feasible vectors pi. The 
combinatorial complexity of determining the integer-valued minimizer of (115) is 
reflected by the complexity of the marginal polytope Mg. This complexity is due to 
the interaction of variables €;, £; as defined by the edges ij € E of the underlying 
graph, besides the integer constraints £; € [m;], Wi € [n]. 

Formulation (121) is the starting point for convex relaxations by optimizing over 
simpler polytopes, defined by a subset of inequalities that specify facets of Mg. The 
recent paper [92] reports a comprehensive evaluation of a broad range of approaches 
to problem (121). Figure 15 illustrates an application to optical flow estimation. 

While research on inference methods for graphical models is supporting the 
design of new approaches to optical flow estimation, the need to restrict the range 
of u to a finite set U/ is a significant restriction. As a consequence, approaches either 
exploit prior knowledge about u, so as to enable a covering of the relevant range of 
u with high resolution through the set / with bounded size |U/|, or solve problem 
(117) once more after refining U/, based on a first estimate of u. 

For instance, the work [94] exploits the probabilistic model (118) in order 
to estimate locally the uncertainty of a first estimate u, which in turn is 
used to refine the set U¢ so as to accommodate the discretization to the local 
variability of u(x). The approach [95] first determines a coarse estimate of 
u in a preprocessing stage by global phase-based correlation, followed by 
defining possible refinements of u(x) in terms of U. The authors of [96] 
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rely on a prior estimate of the fundamental matrix F (34) using standard 
methods, which enables to properly define / based on the epipolar constraint 
(38). 

In a way, while the former two approaches mimic range refinement of variational 
methods through representing u at multiple scales (section “Multiscale’”), the latter 
approach exploits geometrical prior knowledge in a similar way to variational 
methods as discussed in section “Geometrical Prior Knowledge.” Future research 
during the next decade will have to reveal more clearly the pros and cons of these 
related methods. 


Variational Image Registration 


The objective of image registration is to assign two images in terms of a diffeomor- 
phism u: 2 — 82 of the underlying domain. A major motivation for this inherent 
smoothness of u has been applications to computational anatomy [97], based on 
fundamental work of Grenander, Dupuis, Trouvé, Miller, Younes, and others — 
cf. [97-100] and references therein. 

Another basic motivation for the methodology of image registration is the 
use of point features, so-called landmarks, for establishing sparse assignments, 
which need to be interpolated in a subsequent step to obtain a transform of the 
entire underlying domain. This is usually accomplished by kernel functions that 
span a corresponding Hilbert space of smooth functions with bounded point- 
evaluation functional [101, 102]. Interpolation with thin-plate splines is a well- 
known example, and extensions to approximating deformations are straightfor- 
ward. See [103, 104] for corresponding overviews in connection with medical 
imaging. 

The large deformation diffeomorphic metric matching (LDDMM) approach [99, 
100, 105] that emerged from the works cited above has evolved over the years into 
a theoretical and computational framework for diffeomorphic image registration. In 
particular, the application to the assignment of point sets, in connection with kernel 
functions, leads to a canonical system of ODEs whose numerical solution generates 
a diffeomorphic assignment along a geodesic path on the diffeomorphism group. 
See [106] for recent references and an extension for better handling deformations at 
multiple scales. 

The importance of this framework is due to the well-developed mathematical 
basis and due to its broad applicability in the fields of computational anatomy 
and medical imaging. The mathematical relations to continuum and fluid mechan- 
ics and the corresponding relevancy to imaging problems with physical prior 
knowledge (cf. section “Physical Prior Knowledge’’) are intriguing as well. In the 
field of computer vision, deformable shape matching constitutes a natural class 
of applications, unlike the more common optical flow fields in natural videos 
that typically exhibit discontinuities, caused by depth changes and independently 
moving objects. 
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5 Open Problems and Perspectives 
Unifying Aspects: Assignment by Optimal Transport 


The mathematical theory of optimal transport [107, 108] provides a general 
formulation of the assignment problem that bears many relations to the approaches 
discussed so far. 

Consider again the setup discussed in section “Assignment by Displacement 
Labeling”: At each location x' indexed by vertices i € V = [n], a vector u(x’) € 
U(x') from a set of candidates U/(x') has to be selected. Put U = Ujenjl/(x'). 
Denote by V’ the index set of all locations {x + U(x!) }a(xiyeut(xi)s Vi € V, that 
u may assign to the locations indexed by V. Then this setup is represented by the 
bipartite graph G = (V, VV’; E) with edge set E = {ij eV x V’:dueUu, x'+u= 
x/}. The first term of the objective (115) specifies edge weights y; (¢;;U/) for each 
edge corresponding to the assignment x’ + ue, = x/, and minimizing only the first 
term >; <y gi(€:;;U) would independently select a unique vector u(x') from each 
set U(x‘), i € V, as solution to (120). 

A classical way to remove this independency is to require the selection of non- 
incident assignments, that is, besides uniquely assigning a vector u € U(x!) to 
x', Vi € V, it is required that there is at most one correspondence x! + u = x/ 
for all 7 € V’. This amounts to determining an optimal weighted matching in the 
bipartite graph G = (V, V’; E). Formally, collecting the edge weights 9; (¢;;/) by 
a vector 8 € R”, m = 0; <y mi, with subvectors given by (119), the LP 


min (0,4) subjectto w>0, Bew<1jyuv), Be € {0, [yIVUPIXEL, 


peR!E@| 
(122) 
has to be solved where Bg is the incidence matrix of graph G. It is well known 
that the polyhedron RO N {u: Be < Lyuy"} is integral [109], which implies a 
binary solution 7z € {0, 1}!#! to (122) satisfying the required uniqueness condition. 
Note that this condition may be regarded as a weak regularity condition enforcing a 
minimal degree of “smoothness” of the assignment field w. 
The connection to optimal transport can be seen by reformulating problem (122). 
Put n’ = |V’| and let the matrix c ¢€ R’™”” encode the costs of assigning 
(transporting) location x! to x/ = x' + u, u € U(x’). Then consider the problem 


min (c,) subjectto u>0, wl, =1,, w'1, <1y,n<n', (123) 
peRr™n” 


where the unknowns are deliberately denoted again by yu. The second constraint 
says that each node i € V (location x’) is uniquely assigned to some node j € V’ 
(location x/). The third constraint says that at most one vertex i € V is assigned to 
each j € V’. The last condition n’ > n naturally holds in practical applications. It 
is straightforward to show [110, Prop. 4.3] that the solution @ € {0, pre’ to (123) 
is again integral. 
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In the case n = n’, problem (123) equals the linear assignment problem, which 
is a discrete version of Monge-Kantorovich formulation of the optimal transport 
problem. The constraints of (123) then define the Birkhoff polytope, and the 
minimizer j4 at some vertex of this feasible set is a permutation matrix that uniquely 
maps V and V’ onto each other. Matrices jx that are not vertices (extreme points) 
of the polytope are doubly stochastic; hence rows jt; € A,, i € [n] and columns 
[te,;, J € [n] represent nondeterministic assignments of vertices i € V and j € V’, 
respectively. 

The general formulation [107] considers Polish probability spaces (4, 1x), 
(Y, “y) with Borel probability measures ux € P(X), Uy € P(Y), and the set of 
coupling measures, again deliberately denoted by j1, that have jz, [4 as marginals, 


M(x, by) = {we P(X x Y):u(A x V) = wx(A), W(X x B) = wy(B), 


VAC B(X), VBC BYY)}. 
(124) 
Given a Borel cost function c: ¥ x VY + RU {+00}, the problem analogous to (123) 
in the case n = n’ reads 


inf ) c(x, y)du(x, y). (125) 
HEM (Lx Hy) JXxy 


A central question concerns conditions on c that imply existence of deterministic 
minimizers jt of (125), that is, existence of a measurable function T:¥ — Y 
such that for random variables (X, Y) with law jz, the relation Y = T(X) holds. 
The assignment T is called transportation map that “pushes forward” the “mass” 
represented by 1x onto fy, commonly denoted Ty: 


Tax =by with = py(B) = x(T'(B)), VB € BO). (126) 


Likewise, #4 is concentrated on the graph of 7, akin to the concentration of 
minimizers of (123) on a set of binary matrices. 

Due to its generality formulation, (125) provides a single framework for address- 
ing a range of problems, related to optical flow estimation by assignment. This 
particularly includes: 


¢ The representation of both discrete and continuous settings, as sketched above, 
and the applicability to the assignment of arbitrary objects, as defined by the 
spaces ¥, Y. 

¢ The focus on the combinatorial nature of the assignment problem, on convex 
duality and tightness or lack of tightness of the convex relaxation (125), together 
with a probabilistic interpretation in the latter case. 

¢ Conservation of mass reflects the invariance assumption underlying (2) and (6), 
respectively. 

¢ The differential, dynamic viewpoint: Let ¥ = RR“ and define the cost function 
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e(x, y) = |x — yl? (127) 


and the Wasserstein space (Po (X), W>) of Borel probability measures 


P(X) = {ux € P(X): I Ix — yIPdux(x) <00, Vy eX}, (128) 


equipped with the Wasserstein distance 


Wo(x, My) = (/ |x — ylPamx.y)) VE solving (125), 
XXX 


(129) 
with jy replaced by jz’y in (125). Then the path (4,) defined by 


xs =(A—11 +1T),ux (130) 


and some optimal map T via (126) satisfies the continuity equation 
d 
ae + div(v;;) = 0 (131) 


with velocity field v;;R? > R¢@ given by y, = (T—I)o (1-1 + tT)", 
Vt in the sense of distributions. Equation (131) provides a natural connection 
to continuum and fluid mechanics (cf., e.g., [111]) and also to flows generating 
diffeomorphic assignments under additional assumptions [100, Ch. 11]. Com- 
paring (131) and (58) shows that, if g is regarded as a density for the scalar case 
p = 1, condition (57) is a strong assumption implying divu = 0. 


The generality of this framework explains too, however, why the regularity of 
solutions to the Monge-Kantorovich problem is a subtle issue, even when given 
as deterministic assignment 7. This is also apparent through Euler’s equation (131), 
which lacks any viscous term that would induce some regularity. 

From this viewpoint, much of the research related to variational optical flow 
estimation, and to the related problems discussed in Sect. 4, can be understood as: 


(i) Interplay between modeling additional terms that induce a desired degree of 
spatial regularity 

(ii) Investigation of how this affects relaxation of the assignment problem from the 
optimization point of view and the accuracy of its solution. 


As a consequence, no sharp boundaries can (and should) be defined that separate 
these subfields of research. For instance, 
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— The paper [112] suggested an early heuristic attempt to combine bipartite graph 
matching and thin-plate spline-based registration. 

— The work [113] combines smoothing with radial basis functions and MRF-based 
labeling (section “Assignment by Displacement Labeling”) for medical image 
registration. 

— More generally, concerning image labeling, modeling spatial context by the edge- 
indexed terms gj; of the objective (115) entails the need to relax combinatorially 
complex polyhedral feasible sets like the marginal polytope in (121), whose 
vertices may not correspond to deterministic assignments, unlike assignments 
as solutions in the simpler case (122). 

— The authors of [114] introduce a smoothing operator to solve numerically the 
Monge-Kantorovich problem. 

— In [115] a related objective from continuum mechanics is proposed that, for a 
limiting value of some parameter, models a viscous fluid, hence ensures spatial 
regularity in a physically plausible way, as opposed to the pure continuity 
equation (131) that is lacking any such term. Assignments are computed by 
numerically tracing corresponding geodesic paths. 

— Much more general objectives for assignments are addressed in [116] that take 
explicitly into account the metric structure of the underlying space VY. The 
problem to “linearize” this combinatorially complex objective in terms of the 
Monge-Kantorovich problem is studied in [110], along with the problem to 
define a cost function c so as to preserve the discriminative power of the original 
objective as much as possible. 

— The recent work [117] exploits the Wasserstein distance (129) so as to solve 
simultaneously template-based assignment and image segmentation, by globally 
minimizing a corresponding joint variational objective. 


This sample of the literature suggests to conclude that in the field of variational 
image registration (e.g., [100, 115]), sophisticated variational approaches exist 
that are satisfying in both respects (i),(ii) discussed above: these approaches 
clearly exhibit their properties mathematically, and they induce regularity without 
compromising accuracy of assignments, due to a good agreement with the physical 
properties of the objects being matched. 

Outside these fields, a similar quality only holds for variational approaches to 
optical flow estimation that are constrained by — again: physically motivated — state 
equations (section “Physical Prior Knowledge’). A similar level of rigor has not 
been reached yet in a major application area of optical flow estimation: motion- 
based analysis of videos of unrestricted scenes with uncontrolled viewpoint changes 
and with independently moving rigid and articulated objects. This deficiency of 
related models is aggravated by the need for natural extensions of frame-to- 
frame assignments to the permanent analysis of dynamic scenarios over time 
(cf. section “Probabilistic Modeling and Online Estimation’). 
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Motion Segmentation, Compressive Sensing 


Research on compressive sensing [118, 119] and corresponding applications has 
been pervading all fields of empirical data analysis, including image reconstruction 
and more recently video analysis. A central theme is provable guarantees of signal 
recovery in polynomial runtime using sub-Nyquist sampling rates and convex 
relaxations of combinatorial objective functions for signal reconstruction. For 
instance, the most common scenario concerns the recovery of u € IR” fromm <n 
linear measurements Au = b € R”, by minimizing 


min |||; subject to Au = b, (132) 


under the assumption that u is k-sparse, i.e., 
lullo == |supp()| = [ti € fr]: ui A 0} <k. (133) 


The objective in (132) constitutes a convex relaxation of the combinatorial objective 
||\o, and suitable conditions on A, e.g., A is close to an isometry on the subset of 
2k-sparse vectors, guarantee unique recovery of u with high probability. 

This section presents next an extension of this basic reconstruction principle to 
video analysis by sketching the recent work reported by [120]. Let 


fer. relrl, (134) 


denote the raw image sequence data in terms of vectorized image frames f;, t = 
1,2,...,7. Assuming a stationary camera as in surveillance applications, the 
objective is to separate the static background from objects moving in the foreground. 
The ansatz is based on the following modeling assumptions: 


¢ At each point of time ¢ € T, image data are only sampled on a subset 92, C 92 
of the discretized domain 2, resulting in subvectors 


Fa,: teé([T]. (135) 


The sample set $2, may vary with ¢. 

¢ The variation of components of fg, corresponding to the static background is 
caused by global smooth illumination changes. Hence, this part of fo, can be 
represented by a low-dimensional subspace 


Ug. ie Uo, € RIM, tt € [T], (136) 
generated by ny orthonormal columns of a matrix U; that are subsampled on 2; 


and some coefficient vector v;. Research in computer vision [121, 122] supports 
this subspace assumption. 
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¢ Objects moving in the foreground cover only small regions within §2. Hence they 
can be represented by vectors 


SQ, with |supp(s)| <n. (137) 
Putting all together the model reads 
fa, =Uayu+sa,, t€[T], (138) 


and convex relaxation of minimizing |supp(s)| due to (137) leads to the recovery 
approach 


min |[sq, |l1 subject to Ug,vi + Se, = fa,- (139) 
UV; S24 


Comparison to (132) shows similar usage of the sparsity-inducing €! norm and 
subsampled measurements (135) as input data. On the other hand, the low- 
dimensional representation (136) of the static part of the video is estimated as 
well, and the entire video is recovered in terms of U; (hence U rather than Ug, 
is optimized in (139)). In fact, this joint optimization problem is non-convex and 
handled in [120] by alternating optimization: 


— For fixed U;, problem (139) is solved by applying ADMM (cf. section “Non- 
smooth Convex Functionals’’) to the augmented Lagrangian L,(U, v7, Se,, we, ) 
with multiplier vector wg, and parameter A as in (95). 

— Having determined v;, 59,,wg,, the subspace U; is tracked by performing gra- 
dient descent with respect to L(-, v;, 5@,,we,) on the Grassmannian G(ny, R") 
(cf., e.g., [123]), resulting in U;+1. 


The closely related static viewpoint on the same problem reveals its relevancy to 
several important research directions. Let 


F=([fi,...,frl=L+8 (140) 
denote the whole video data that, due to the reasoning above, are supposed to be 


decomposable into a Jow-rank matrix L and a sparse matrix S. The corresponding 
convex relaxation approach [124] reads 


min |L]l<x + aS subject to L+S =F, (141) 
where ||L||.« = > 0; 0;(L) denotes the nuclear norm in terms of the singular 
values of L and ||S||; = >0;,; |Sij|. Here, the nuclear norm ||- || constitutes a 


convex relaxation of the combinatorial task to minimize the rank of L, analogous 
to replacing the combinatorial objective ||u||o in (133) by ||u||, in (132). Clearly, the 
online ansatz (138) along with the corresponding incremental estimation approach 
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is more natural for processing long videos. The price to pay is the need to cope with 
a non-convex (albeit smooth) problem, whereas the batch approach (141) is convex. 

Future research will tackle the challenging, more general case of non-static 
backgrounds and moving cameras, respectively. For scenarios with small displace- 
ments u(x), work that represents the state of the art is reported in [125]. Results 
in computer vision that support subspace models and low-rank assumptions have 
been established [126], and the problem of clustering data lying in unknown low- 
dimensional subspaces has received considerable attention [127-129]. 

From a broader perspective, video analysis and motion-based segmentation 
provide attractive connections to research devoted to union-of-subspaces models 
of empirical data and relevant compressive sensing principles [130-132] and 
to advanced probabilistic models and methods for nonparametric inference 
[133, 134]. 


Probabilistic Modeling and Online Estimation 


There is a need for advanced probabilistic models, and three related aspects of 
increasing difficulty are briefly addressed: 


¢ A persistent issue of most variational models of mathematical imaging, including 
those for optical flow estimation, concerns the selection of appropriate hyper- 
parameter values, like the parameter o of (62) weighting the combination of 
data term and regularizer (56). In principle, Bayesian hierarchical modeling 
[135] provides the proper framework for calibrating variational models in this 
respect. The paper [136] illustrates an application in connection with optical 
flow estimation, based on the marginal data likelihood [137] interpreted as 
hyperparameter (model) evidence. 

Estimating hyperparameter values from the given data in this way entails 
the evaluation of high-dimensional integrals for marginalization, commonly 
done using Laplace’s method and a corresponding approximation by Gaussian 
(quadratic) integrals [138, 139]. A validation for complex high-dimensional 
posterior distributions encountered in variational imaging is involved, however, 
and is also stimulating more recent research in the field of statistics [140]. 

Using discrete variational models (section “Assignment by Displacement 
Labeling”) aggravates this problem, due to considerable computational costs 
and since no widely accepted methods have been established analogous to the 
abovementioned approximations. 

¢ Computational costs in connection with runtime requirements become a serious 
problem when dynamic scenarios are considered. While extensions of the 
domain to 2 x [0, T] like in (82) are straightforward mathematically and have 
proven to significantly increase accuracy of optical flow estimation, employing 
a static model in terms of elliptic Euler-Lagrange systems to a dynamic system 
appears somewhat odd, not to mention the need to shift the time interval [0, 7] 
along the time axis in order to analyze long image sequences. 
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Such extensions appear more natural in connection with dynamic physical 
models constraining optical flow estimation, as opposed to stationary formula- 
tions like (88). See [141] for a corresponding approach to data assimilation [142]. 
A nice feature of this method is the ability to estimate initial conditions that are 
generally unknown, too. On the other hand, the computational costs necessitate to 
propagate a low-dimensional POD-projection of the state variables (POD: proper 
orthogonal decomposition) since the control of dynamical systems [70] entails 
looping forward and backward through the entire time interval. 

¢ The last remark points to the need for online estimation methods that are causal 
and optimal, in connection with the analysis of dynamical system through image 
analysis. Again the proper framework is known since decades: Given stochastic 
state and observation processes 


S = {Si}i>0, G = {Gr}is0, (142) 


stochastic filtering[143] amounts to determine the conditional distribution of S; 
given the observation history and to evaluate it in terms of expectations of the 
form E[g(S;)|g;, 0 < s < t], for some statistic g(-) of interest (e.g., simply 
y(S;) = S;) and conditioned on realizations g, of G;, s € [0,t]. Most research 
during the last decade considered the design of particle filters [143, 144] to the 
estimation of /ow-dimensional states based on image measurements. This does 
not scale-up however to high-dimensional states like optical flows S; = up. 

An attempt to mimic online estimation in connection with instationary optical 
flows related to experimental fluid dynamics is presented in [68], with states 
and their evolution given by vorticity transport. For low signal-to-noise ratios 
and sufficiently high frame rates, the approach performs remarkably well. 
Another dynamical computer vision scenario is discussed in the recent work 
[57]. Here the states S, = (z;,{h;, R;}) € IR” x SE(3) are dense depth-maps 
z (cf.(83)) together with varying motion parameters {h,, R,} describing the 
observer’s motion relative to the scene, to be estimated from image sequence 
features g; aS measurements via optical flow estimates u,; — see Fig. 13. The 
approach involves prediction and fusion steps based on Gaussian approximation 
and joint optimization, yet cannot be considered as direct application of the 
stochastic filtering framework, in a strict sense. This assessment applies also to 
labeling approaches (section “Assignment by Displacement Labeling”’) and their 
application to dynamic scenarios. 


6 Conclusion 


Optical flow estimates form an essential basis for low-level and high-level image 
sequence analysis and thus are relevant to a wide range of applications. Corre- 
sponding key problems, concepts, and their relationships were presented, along with 
numerous references to the literature for further study. Despite three decades of 
research, however, an overall coherent framework that enables to mathematically 
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model, predict, and estimate the performance of corresponding computational 
systems in general scenarios is still lacking. This short survey will hopefully 
stimulate corresponding methodological research. 


7 Basic Notation 


List of major symbols used in the text 


Symbol Brief description Reference 
r.h.s. abbr.: right-hand side (of 
some equation) 
w.r.t. abbr.: with respect to 
w.l.o.g. without loss of generality 
LP linear program 
1, € R" Ce eres 0) 
[n], ne IN integer range {1,2,...,n} 
[n]o, n € IN integer range 
{0,1,...,2—1} 
2c R! image domain; typically 
d € {2,3} 
x= (x1.....xa) € 2 | image point 
u(x,t) € Ré assignment, motion or (4), (7), (8) 


X = (Xj, X2, X3)' € 
R? 
,er Yer 


SO(3), 50(3) 
SE(3) 
{h, Rt € SE(3) 


[d]x € s0(3), g € R® 


optical flow field 


Sections “Assignment 
Fields” 


and “Motion Fields” 
scene point 


Section “Two-View 
Geometry” 


homogeneous 
representation of 

image and scene points x 
and X, resp. 

special orthog. group and 
its Lie algebra 

group of Euclidean (rigid) 
transf. of IR? 

Euclidean transformation 
of R? 

skew-symm. matrix 
defined by 


(28) 


(continued) 
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Symbol 


KeR*® 
F,E eR? 


f(x,t), xe 2, teR 


=. -F 
0; => xi" LE [d] 
e ~ or lo| 
a ole 
0 — te: 
ae’, 
la| = Viet Qi 
w* — wr! won 


aoe 
Vif (e.0) = (F4E3 ) 
divu 


A 


g(x,t)EeR’, p>=1 


Jg(x) = 

(vs: oe eee 

Jei(x,t) = 
Vigi(x.t)),) je 

(Vs i) sa, 

&(@) = Fg) = 

(Fg)(o) 

(x, x’) = Do XiX; 

lx =x)? 

Ilacl]i = D0; lil 

diag(x) 

ker A 

trA = es Ajj 


oe". 
1 f (x,t) 
Vf (x,t) = ( : ) 


Brief description 

Iq)xX =qxX, VX € 
R3 

camera matrix (internal 
parameters) 


fundamental and essential 


matrix 
image sequence 
spatial partial derivative 


temporal partial derivative 


multi-index notation 


monomial from w € RZ 


spatial gradient 


spatio-temporal gradient 


divergence Yield] 0, uj; of 


a vector field u 
Laplace operator 


Viet] a; 


feature mapping (specific 


meaning 
and p depend on the 
context) 


Jacobian matrix of 
g(x) € R? atx € R4 


Jacobian of g(x, t) € R? 
at (x,t) € Rt! 


Fourier transform of g 


Euclidean inner product 
Euclidean £7 norm 

! norm 

diagonal matrix with 
vector x as diagonal 
nullspace of the linear 
mapping A 

trace of matrix A 


1997 


Reference 


Section “Two-View 
Geometry” 
Section “Two-View 
Geometry” 


page 1952, (17) 


(continued) 
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Symbol Brief description Reference 
(A, B) = tr(A'B) matrix inner product 
| Alle = (A, A)!/? Frobenius norm 
p:R > Ry distance function page 1948, (3) 
6c(x) = indicator function of a 
0, xeCc 
too, x ¢#C 
closed convex set 
ce R? 
IIc orthogonal projection 
onto 
a closed convex set C 
A, C R" probability simplex page 1985, (119) 
{x ERY ep Xi = 
1; x > 0} 


Cross-References 


Compressive Sensing 

Duality and Convex Programming 
Energy Minimization Methods 

Iterative Solution Methods 

Large-Scale Inverse Problems in Imaging 
Linear Inverse Problems 

Non-Linear Image Registration 
Regularization Methods for Ill-Posed Problems 
Splines and Multiresolution Analysis 
Statistical Methods in Imaging 

Total Variation in Imaging 
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Abstract 

Image registration is to automatically establish geometrical correspondences 
between two images. It is an essential task in almost all areas involving imaging. 
This chapter reviews mathematical techniques for nonlinear image registration 
and presents a general, unified, and flexible approach. Taking into account that 
image registration is an ill-posed problem, the presented approach is based 
on a variational formulation and particular emphasis is given to regularization 
functionals motivated by mathematical elasticity. Starting out from one of the 
most commonly used linear elastic models, its limitations and extensions to 
nonlinear regularization functionals based on the theory of hyperelastic materials 
are considered. A detailed existence proof for hyperelastic image registration 
problems illustrates key concepts of polyconvex variational calculus. Numerical 
challenges in solving hyperelastic registration problems are discussed and a 
stable discretization that guarantees meaningful solutions is derived. Finally, two 
case studies highlight the potential of hyperelastic image registration for medical 
imaging applications. 


1 Introduction 


Image registration is an essential task in a variety of areas involving imaging 
techniques such as astronomy, geophysics, and medical imaging; see, e.g., [11, 
27, 31, 45, 52, 55, 73] and references therein. The goal of image registration is 
to automatically find geometrical correspondences between two or more images 
acquired; see also Fig. 1 for a simplified example. Measurements may result from 
different times, from different devices, or perspectives. More specifically, one 
aims to find a reasonable transformation, such that a transformed version of 
the first image is similar to the second one. Mathematically, image registration 
is an ill-posed problem and is thus typically phrased as a variational problem 
involving data fit and regularization. The data fit or distance functional measures the 
similarity of the images. The regularization functional quantifies the reasonability 
of the transformation and can also be used to guarantee a mathematically sound 
formulation and to favor solutions that are realistic for the application in mind. 

This chapter presents a comprehensive overview of mathematical techniques 
used for nonlinear image registration. A particular focus is on regularization 
techniques that ensure a mathematically sound formulation of the problem and allow 
stable and fast numerical solution. 

Starting out from one of the most commonly used linear elastic models [10,55], 
its limitations and extensions to nonlinear regularization functionals based on the 
theory of hyperelastic materials are discussed. A detailed overview of the available 
theoretical results is given and state-of-the-art numerical schemes as well as results 
for real-life medical imaging applications are presented. 
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image 7 TJ and transformation y deformed image Ty] =Toy 


Fig. 1 Transforming a square into a disc: template image 7 is a white square on black background 
(left), reference image FR is a white disk on black background (not displayed), the transformation 
y is visualized by showing a regular grid X as overlay on the deformed image T[y] (right) and as 
y(X) on the template image (center); note that T[y](x) = T(y(x)) 


2 The Mathematical Setting 


The purpose of this section is to provide a brief, conceptual introduction to a 
variational formulation of image registration, see section “Variational Formulation 
of Image Registration”. For brevity, the presentation is restricted to a small subset 
of the various configurations of the variational problem and is limited to 3D 
problems which today is most relevant in medical imaging applications. Section 
“Images and Transformation” defines a continuous model for images and formalizes 
the concept of transformations y and transformed images 7[y]. Section “Length, 
Area, and Volume Under Transformation” discusses the transformation of length, 
area, and volume. These geometrical quantities play an important role for the 
computation of locally invertible transformations, which are important for many 
clinical applications. 

After having discussed the general concepts on a rather broad level, specific 
building blocks of the variational framework are examined in more detail. Section 
“Distance Functionals” introduces two common choices of distance functionals D, 
which measures the alignment of the images. After motivating the demand for 
regularization by illustrating the ill-posedness of the variational problem in section 
“Tl-Posedness and Regularization”, an overview of elastic regularization functionals 
and a discussion of their limitations is given in section “Elastic Regularization 
Functionals”. 

Motivated by the limitations of elastic regularization functionals extensions 
to hyperelastic functionals, designed to ensure invertible and reversible trans- 
formations, are presented in section “Hyperelastic Regularization Functionals”. 
Subsequently, section “Constraints” presents two examples that demonstrate how 
constraints C can be used to restrict set of admissible transformations A, to favor 
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practically relevant solutions, and most importantly, to exclude practically infeasible 
solutions. 

Finally, a comprehensive overview of related literature and suggestions for 
further readings are provided in section “Related Literature and Further Reading”. 


Variational Formulation of Image Registration 


Given the so-called template image 7 and the so-called reference image 7, the 
image registration problem can be phrased as a variational problem. Following [55], 
an energy functional 7 is defined as a sum of the data fit D depending on the input 
images and the transformation and a regularization S, 


J ly] := DIT, R: y] + S[y]. (1) 
The objective is then to find a minimizer y of 7 in a feasible set A to be described, 
min J[y] for ye. (2) 


Advantages of this formulation are its great modelling potential and its modular 
setting that allows to obtain realistic models tailored to essentially any particular 
application. 


Images and Transformation 


This section introduces a continuous image model, transformations, and discusses 
geometrical transformations of images. Although the image dimension is conceptu- 
ally arbitrary, the description is restricted to 3D images for ease of presentation. 


Definition 1 (Image). Let 2 C R? be a domain, i.e., bounded, connected, and 
open. An image is a one-time continuously differentiable function 7 : R? > R 
compactly supported in §2. The set of all images is denoted by Img(2). 


Note that already a simple transformation of an image such as a rotation requires 
a continuous image model. Therefore, a continuous model presents no limitation, 
even if only discrete data is given in almost all applications. Interpolation techniques 
are used to lift the discrete data to the continuous space; see, e.g., [55] for an 
overview. As registration is tackled as an optimization problem, continuously 
differentiable models can provide numerical advantages. 


Definition 2 (Transformation). Let 2 C R° be a domain. A transformation is a 
function y : 2 — R?. For a differentiable transformations, the Jacobian matrix is 
denoted by 
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O11 O2y1 O3N1 
Vyi= (ay | doy | day) := [| O12 d2y2 0392 
d1y3 0293 0393 


The function u : 2 — R? satisfying y(x) = x + u(x) is called displacement, 
Vy =I1I+Vy, where I denotes a diagonal matrix of appropriate size with all ones 
on its diagonal. 


Example I. An important class of transformations are rigid transformations. Let 
b € R? describes a translation and Qe R?*3 a rotation matrix, 


Arigia (2) := {y v(x) = Ox+bae.ony, 
beR’, QE R™, O'O =I, dettQ = 1}. 


A transformation y is rigid on X, if y € Arigia(’). 


Given an image 7 and an invertible transformation y, there are basically the 
Eulerian or Lagrangian way to define the transformed version 7 [y]; see [38,55] for 
more details. In the Eulerian approach the transformed image is defined by 


Th]: RR, Tp) := TO)). (3) 


In other words, assigning position x in J [y] the intensity of 7 at position y(x); see 
illustration in Fig. 1. The Lagrangian approach transports the information (x, 7 (x)) 
to (v(x), 7 (x)), where the first entry denotes location and the second intensity. 
In image registration, typically, the Eulerian framework is used. However, the 
Lagrangian framework can provide advantages in a constrained setting; see [38] 
for a local rigidity constrained and the discussion in section “Constraints”. 


Length, Area, and Volume Under Transformation 


This section relates changes of length, area, and volume in the deformed coordinate 
system to the gradient of the transformation y. To simplify the presentation, it is 
assumed that y is invertible and at least twice continuously differentiable. 

Using Taylor’s formula [18], a local approximation of the transformation at an 
arbitrary point x in direction v € R? is given by 


y(x + v) = yx) + Vy) v + O(\v)?). (4) 
Choosing v = he;, it can be seen that the columns of the gradient matrix Vy(x) 


approximate the edges of the transformed reference element; see also Fig. 2. Thus, 
changes of length, area, and volume of a line segment L connecting x and x + he;, 
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Fig. 2 Visualization of a deformation of a tetrahedron and the Taylor approximation of the 
transformation y. At an arbitrary point x a tetrahedron is spanned by the scaled Euclidean unit 
vectors (left). The edges of the deformed tetrahedron are approximated by the columns of the 
gradient V y(x) (right). Due to this approximation property, geometric quantities such as volume, 
area, and length can be related to the gradient matrix, its cofactor, and its determinant 


a triangle T spanned by x, x + he;, and x + hex, and a tetrahedron P spanned by 
x and the three scaled Euclidean vectors are approximately 


length(y(L)) = |y(x + he;) — y(x)| = |d;y| + OM’), 
h2 
area(y(7T)) = 5 |ajy x ax y| + O(h’), 


vol(y(P)) = h? det (diy | doy | sy) + O(h?) = h det Vy + O10), 


where x denotes the outer product in R?. 
The cofactor matrix cofVy summarizes, column wise, the area changes of the 
three triangles spanned by x with two scaled Euclidean vectors each, 


cofVy = (ay x 03 | d3y X Oy | dry x doy) er’, 


At this point, the importance of limiting the range of det Vy for many clinical 
applications should be stressed. Due to the inverse function theorem [25, Sect. 5], a 
transformation y € C!(2, R°) is locally one-to-one if det Vy #4 0. From the above 
considerations, it can be seen that det Vy = 0 indicates annihilation of volume and 
det Vy < 0 indicates a change of orientation. Thus, controlling the range of det Vy, 
or analogously the range of compression and expansion of volume, ensures local 
invertibility and thus is of utmost importance in practical applications. 


Distance Functionals 


The variational formulation of the image registration problem (2) characterizes an 
optimal transformation as a minimizer of the sum of a distance and a regularization 
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functional subject to optional constraints. These three building blocks will be 
introduced and discussed in the following sections. 

Distance functionals mimic the observer’s perception to quantify the similarity of 
images. In the following, two examples of distance measures will be presented and 
the reader is referred to [55] for further options. Emphasis is on a mass-preserving 
distance measure, which is an attractive option for the registration of density images 
such as Positron Emission Tomography (PET) images; see also section “Motion 
Correction of Cardiac PET”. 

A robust and widely use distance measure is the L?-norm of the intensity 
differences commonly referred to as the sum-of-squared-difference (SSD), 


DSP Ly] = DSPIT,R; y] = 5 / (T(x) — RQ@))Pdx. (5) 
Q 


For this distance measure to be effective, intensity values at corresponding points 
in the reference and template image are assumed to be equal. This assumption is 
often satisfied when comparing images of the same modality, commonly referred 
to as mono-modal registration. Hence, DSS? is a prototype of mono-modal distance 
functionals. 

In some applications intensities at corresponding points differ conceptually. 
Typical examples of multimodal image registration include the fusion of different 
modalities like anatomy (such as CT or MRI) and functionality (such as SPECT or 
PET); see, e.g., [55] for further options and discussions. 

Mass densities are another example for conceptional different image intensities. 
The necessity of mass-preserving transformations was first discussed in [29]. 
Recently, mass-preserving distance functionals were used to register images from 
Positron Emission Tomography (PET); see section “Motion Correction of Cardiac 
PET” and [14, 30, 60]. Another application is the correction of image distortions 
arising from field inhomogeneities in Echo-Planar Magnetic Resonance Imaging; 
see section “Susceptibility Artefact Correction of Echo-Planar MRI” and [15,62]. 

Due to mass-preservation, change of volume causes change of intensities. Note 
that the simple model for transformed images (3) does not alter image intensities. 
Let V C §2 denotes a small reference volume. Ideally, the mass of R contained in 
V has to be equal to the mass of 7 contained in y(V). Formally, 


/ R(x)dx = T(x)dx = / T (y(x))- det Vy(x) dx, (6) 
Vv y(V) Vv 


where the second equality holds by the transformation theorem [18, p. 31f], assum- 
ing that the transformation is injective, continuously differentiable, orientation 
preserving, and that its inverse is continuous. A natural mass-preserving extension 
of the SSD distance functional thus reads 


1 
DML] := 5 / (T(y(@))- det Vy (x) — R(X)? dx. (7) 
2 
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Ill-Posedness and Regularization 


A naive approach to image registration is to simply minimize the distance functional 
D with respect to y. However, this is an ill-posed problem [23,27,46,55]. According 
to Hadamard [41], a problem is well-posed, if there exists a solution, the solution is 
unique, and the solution depends continuously on the data. If one of these criteria is 
not fulfilled, the problem is called ill-posed. 

In general, existence of a minimizer of the distance functional cannot be 
expected. Even the rather simple SSD distance functional (5) depends in a non- 
convex way on the transformation; for a discussion of more general distance 
functionals, see [55, p. 112]. If the space of plausible transformations is infinite 
dimensional, existence of solutions is thus critical. 

A commonly used approach to bypass these difficulties and to ensure existence 
is to add a regularization functional that depends convexly on derivatives of the 
transformation [10, 67, 68]. This strategy is effective for distance functionals are 
independent of or depend convexly on the gradient of the transformation. This is 
the case in most applications of image registration, for instance, for the SSD in (5). 
However, further arguments are required for the mass-preserving distance functional 
in (7) due to the non-convex of the determinant; see Sect. 1. 

It is also important to note that distance functionals may be affected considerably 
by noise; see also [55]. This problem is often (partially) alleviated by using regu- 
larization functionals based on derivatives of the transformation. This introduces 
a coupling between adjacent points, which can stabilize the problem against such 
local phenomena. 

Note, even though regularization ensures existence of solutions, the objective 
functional depends in a non-convex way on the transformation and thus a solution 
is generally not unique as the following example illustrates. 


Example 2. Consider an image of a plain white disc on a plain black background as 
a reference image and a template image showing the same disc after a translation. 
After shifting the template image to fit the reference image, both images are identical 
regardless of rotations around the center of the disc. Hence, there are infinitely many 
rigid transformations yielding images with optimal similarity. Regularization can be 
used to differentiate the various global optimizers. 


Elastic Regularization Functionals 


The major task of the regularization functional S is to ensure existence of solutions 
to the variational problem (2) and to ensure that these solutions are plausible for the 
application in mind. Therefore, regularization functionals that relate to a physical 
model are commonly used. In this section it is assumed that the transformation y is 
at least one time continuously differentiable and sufficiently measurable. A formal 


Non-linear Image Registration 2013 


definition of an elastic regularization functional and its associated function spaces 
is postponed to section “Hyperelastic Regularization Functionals”. 

The idea of elastic regularization can be traced back to the work of Fischler 
and Elschlager [28] and particularly the landmarking thesis of Broit [10]. The 
assumption is that the objects being imaged deform elastically. By applying an 
external force, one object is deformed in order to minimize the distance to the second 
object. This external force is counteracted by an internal force given by an elastic 
model. The registration process is stopped in an equilibrium state, that is, when 
external and internal forces balance. 

In linear elastic registration schemes, the internal force is based on the displace- 
ment u and the Cauchy strain tensor 


V = V(u) = (Vu Vu')/2, 


which depends linearly on u. The linear elastic regularization functional is then 
defined as 


S*5[y] = [ v(trace V)? + ptrace(V*) dx, 


where v and y are the so-called Lamé constants [18,53]. 

Benefits and drawbacks of the model S®"* relate to the linearity of V in Vu. Its 
simplicity has been exploited for the design of computationally efficient numerical 
schemes such as in [37]. A drawback is the limitation to small strains, that is, 
transformations with ||Vu|| < 1; see [10, 18,55]. For large geometrical differences, 
the internal forces are modelled inadequately and thus solutions may become 
meaningless. 

While the motivation in [10] is to stabilize registration against low image quality, 
elastic regularization also ensures existence of solutions in combination with many 
commonly used distance functionals. Another desired feature of this regularizer is 
that smooth transformations are favored, which is desirable in many applications. 

In order to overcome the limitation to small strains, Yanovsky et al. [72] proposed 
an extension to nonlinear elasticity. They used the Green-St.-Venant strain tensor 


E = E(u) = (Vu+ Vu! + Vu' Vu)/2, (8) 


which is a quadratic in Vu. Revisiting the computations of changes in length in 
section “Length, Area, and Volume Under Transformation”, it can be shown that & 
penalized changes of lengths. The quadratic elastic regularization functional of [72] 
reads 


sui] = / v(trace E)* + ptrace(E’) dx. (9) 
2 
While relaxing the small strain assumption, the functional is not convex as the fol- 


lowing example shows. Thus, additional arguments are required to prove existence 
of a solution to the variational problem based on S4, 
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Fig. 3. Non-convexity of 
S4 in (9) also used in 9 |\ | 
Yanovsky et al. [72]. The 


value of St*4 is plotted for = a | 
transformations g 1 7 | 
Ve += x + diag(c,c,0)x, D \ 
c € [-2.5, 0.5] assuming nN 
parameters v = xp = 1 0 N aot XK 

—2 —1 0 


Example 3 (Non-convexity of S***), For c € Ra transformation y, is defined 
as ye(x) := x + diag(c,c,0)x such that u.(x) = diag(c,c,0)x. Thus, E(u.) = 
diag(c + c?/2,c + c*/2,0). Assuming v = jz = 1 in (9), it holds that S®“[u,] = 
6(c + c?/2)?. 

Picking c = 0 and c = —2, it holds S®*4(u_y) = S%**(uy) = 0; see also Fig. 3 
for the landscape of S%*4(u,). The non-convexity of S%*4 with respect to y. can 
then be seen by considering convex combinations of y_, and yo. Since S"4(u_,) = 
3/2 > 0, S%4 is not convex in y. Also note that y_; is non-invertible and still has 
finite energy. 


Another serious drawback of both the energies S*"S and S™*4 is that both yield 
finite energy for transformations that are not one-to-one; see Example 4. In addition, 
even for invertible transformations, the above regularization functionals do not 
explicitly control tissue compression and expansion; see Example 5. Thus, unlimited 
compression is performed, if sufficient reduction of the distance functional can be 
gained. 


Example 4. Consider the non-invertible trivial map y(x) = 0, with u(x) = —x and 
Vu = —I. Clearly, V = 4E = —2I and S*"5[u] = 4S%4[u] = (9v + 3)vol(@). 
These regularizers give thus finite values that depend only on the Lamé constants 
and the volume of the domain £2. 


Example 5. For the sequence of transformations y)(x) := 27"x, it follows 
u(x) = (2 — 1)x and det Vy = 27-3”, For large n, this transformation heavily 
compresses volume. However, the components of Vu are bounded below by —1, 
showing that arbitrary compressions are not detected properly. 


Hyperelastic Regularization Functionals 


Motivated by these observations, a desirable property of regularization functionals 
is rapid growth to infinity for detVy —> Ot. In the Example 5, the Jacobian 
determinant is det Vy = 2°” indicating that the volume of a reference domain 
goes rapidly to zero with growing n. Controlling compression and expansion to 
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physically meaningful ranges has been a central theme in image registration; see, 
e.g., [34-36, 40, 58, 59]. 

Models for the so-called Ogden materials [4, 18] use an energy function that 
satisfies this important requirement, 


1 
SOsdens 5) — [ soelVul? + aal|cofV y ||7- + a, Wo(det Vy)dx, (10) 


with regularization a parameters ay, a,,a@, > 0, the Frobenius norm ||A\|- = 
ay j=1 a? , and the convex penalty Wo(v) = v?—log v. Note that this penalty ensures 
lim,_,o+ Wo(v) = lim,+oo Yo(v) = oo and thus gives infinite energy for non- 
invertible transformations. For small strains, linear elasticity is an approximation 
to this regularization functional; see [18]. For image registration, this energy was 
introduced first by Droske and Rumpf [23]. 

While ensuring invertible transformations, the hyperelastic regularization func- 
tional (10) has drawbacks in the context of image registration: On the one hand, 
setting a, > O puts a bias towards transformations that reduce surface areas. 


On the other hand, setting a, = 0 prohibits the standard existence theory as 
presented in the next section. In addition, Wo is not symmetric with respect 
to inversion, that is, Wo(v) 4 Wo(1/v). Since detVy~! = (detVy)~! the 


value of the volume regularization term changes, when interchanging the role of 
template and reference image; see [17] for a discussion of the inverse consistency 
property. 

To overcome these two drawbacks, the following hyperelastic regularizer as a 
slight modification of the Ogden regularizer (10) was introduced in [14], 


shetty] = Soe c1Vul? + aa(x)pe(cofVy) + a(x) W(det Vy)dx. 
2 


Making the regularization parameter spatially dependent is a small conceptual 
contribution with potentially big impact for many applications. The main idea is 
to replace Wo by a convex penalty y, fulfilling the growth condition, and satisfying 
w(v) = w(1/v). More precisely, the convex functions 


2 
¢e(C) = Pema cH 1,04 and y(v) = =" 


were suggested in [14]. The penalty ¢, is a convexification of the distance to 
orthogonal matrices and the penalty y(v) is essentially a polynomial of degree two, 
thus guaranteeing det Vy € L”, which is important in mass-preserving registration. 
If detVy € L! suffices, w can be replaced by its square root. Note that @. is 
designed to avoid the preference for area shrinkage in (10). A convexification is 
needed for the existence theory, which makes ¢, blind against shrinkage, allowing 
only penalization of area expansion. 
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Constraints 


It is often beneficial to include additional prior knowledge or expectations on the 
wanted transformation into the model. Two case studies will show exemplarily how 
to restrict the set of admissible transformations. The first one is related to volume 
constraints [35, 36,59] and the second one to local rigidity constraints [38, 65]. 

Volume-preserving constraints are particularly important in a setting such as 
monitoring tumor growth, where a change of volume due to the mathematical 
model can be critical. In addition, registration problems requiring highly nonlinear 
deformations, e.g., arising in abdominal or heart imaging, can be stabilized by 
restricting volume change to a certain class. 

Constraining compression and expansion of volume has been a central topic in 
image registration over many years. Following [36], a formal description reads 


Km(x) < CY"[y](x) := det Vy(x) < Ky(x) for x € Q, 


where kK, and ky are given bounds. Volume preservation was enforced by using 
penalties, equality constraints such as Kk, = Ky = 1, or box constraints; see, e.g., 
[34—36, 40, 58,59]. 

Another example for image-based constraints is local rigidity. Local rigidity 
constraints can improve the model in applications in which images show bones or 
other objects that behave approximately rigidly, such as head-neck images, motion 
of prostate in the pelvis cage, or the modelling of joints. Rigidity on a subdomain 
& C 92 can be enforced by setting 


C'R[y, Z] := dist(y, Arigia(Z)}) = 0; 


see also Example |. This formulation can be extended to multiple regions; 
see [38, 65]. 

Based on the discussion in [38], the Lagrangian framework has the advantage 
that the set » does not depend on y and thus does not need to be tracked; 
see [54] for a tracked Eulerian variant. Tracking the constrained regions may add 
discontinuities to the registration problem. In the case of local rigidity constraints, 
constraint elimination is an efficient option and results linear constraints. However, 
the Lagrangian framework involves det V y in the computations of D and S; see [38] 
for details. 


Related Literature and Further Reading 


Due to its many applications and influences from different scientific disciplines, 
it is impossible to give a complete overview of the field of image registration. 
Therefore this overview is restricted to some of the works, which were formative 
for the field or are relevant in the scope of this chapter. For general introductions to 
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image registration, the reader is referred to the textbooks [33, 53,55]. In addition, 
the development of the field has been documented in a series of review articles that 
appeared in the last two decades; see [11,27,31,45,52,73]. 

A milestone for registration of images from different imaging techniques, or 
modalities, was the independent discovery of the mutual information distance 
functional by Viola and Wells [71] and Collignon et al. [19]. In this informa- 
tion theoretic approach, the reference and template images are discretized and 
interpreted as a sequence of grey values. The goal is to minimize the entropy of 
the joint distribution of both sequences. Due to its generality, mutual information 
distance measures can be used for a variety of image modalities. However, it is 
tailored to discrete sequences and thus the mutual information of two continuous 
images is not always well defined; see [42]. Furthermore, a local interpretation 
of the distance is impossible. To overcome these limitations, a morphological 
distance functional [23] and an approach based on normalized gradient fields [56] 
were derived for multimodal registration. Both approaches essentially use image 
gradients to align corresponding edges or level sets. 

In addition to the already mentioned elastic and hyperelastic regularization 
functionals, another example is the fluid registration by Christensen [16]. The 
difference between fluids and elastic materials lies in the model for the inner force. 
In contrast to elastic materials, fluids have no memory about their reference state. 
Thus, the fluid registration scheme allows for large, nonlinear transformations while 
preserving continuity and smoothness. However, similar limitations arise from the 
fact that volume changes are not monitored directly. 

It is well known that nonlinear registration schemes may fail if the geometrical 
difference between the images is too large. To overcome this, a preregistration 
with a rigid transformation model is usually employed. The above regularizers are, 
however, not invariant to rigid transformations. To overcome this limitation, the cur- 
vature regularization functional, based on second derivatives, was proposed [26,43]. 

Similar to the linearized elastic regularization functional, fluidal or curvature 
regularized schemes fail to detect non-invertible transformations. As an alternative 
to hyperelastic schemes, the invertibility of the solution is to restrict the search 
for a plausible transformation to the set of diffeomorphisms as has been originally 
suggested by Trouvé in 1995 [69] and resulted in important works [3,5, 70]. A 
transformation y is diffeomorphic, if y and y~! exist and are continuously differ- 
entiable. The existence of an inverse transformation implies that diffeomorphisms 
are one-to-one. While invertibility is necessary in most problems, it is not always 
sufficient as the Example 5 indicates: for large n the size of the transformed domain 
has no physical meaning; see [55] for further discussions. 

Image registration is closely related to the problem of determining optical flow, 
which is a velocity field associated with the motion of brightness pattern in an image 
sequence. First numerical schemes have been proposed by Horn and Schunck [48] 
and Lucas and Kanade [51] in 1981. In their original formulations, both approaches 
assume that a moving particle does not change its intensity. This gives one scalar 
constraint for each point. Hence, as image registration, determining the flow field 
is an under-determined problem. The under-determinedness is addressed differently 
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by both approaches. The method by Horn and Schunck generates dense and globally 
smooth flow fields by adding a regularization energy. Lucas and Kanade, in contrast, 
suggest to smooth the image sequence and compute the flow locally without 
additional regularization. A detailed comparison and a combination of the benefits 
of both approaches is presented in [12]. 

There are many variants of optical flow techniques. For example, Brune replaced 
the brightness constancy by a mass-conservation constraint; see [13]. This gives the 
optical flow pendant to mass-preserving image registration schemes; see [30, 60, 
61] and sections “Distance Functionals” and “Motion Correction of Cardiac PET”. 
Further, rigidity constraints have been enforced in optical flow settings in [SO]. 

One difference between image registration and optical flow lies in the available 
data. In optical flow problems, motion is typically computed based on an image 
sequence. Thus, the problem is space and time dependent. If the time resolution of 
the image sequence is sufficiently fine, geometrical differences between successive 
frames can be assumed to be small and displacements can be linearized. In contrast 
to that, in image registration the goal is typically to compute one transformation that 
relates two images with typically substantial geometrical differences. 

The problem of mass-preserving image registration is closely related to the 
famous Monge—Kantorovich problem of Optimal Mass Transport (OMT) [1, 6, 24, 
49]. It consists of determining a transport plan to transform one density distribution 
into another. To this end, a functional measuring transport cost is minimized over 
all transformations that satisfy DM? [y] = 0 in (7). A key ingredient is the definition 
of the cost functional which defines optimality. Clearly there is an analogy the 
cost functional in OMT and the regularization functional in mass-preserving image 
registration. 


3 Existence Theory of Hyperelastic Registration 


The variational image registration problem (2) is ill-posed problem in the sense 
of Hadamard; see section “Ill-Posedness and Regularization” and [27,55]. As 
Example 2 of a rotating disc suggests, uniqueness can in general not be expected. 
However, hyperelastic image registration has a sound existence theory which will 
be revisited in this section. 

The main result of this section is the existence theorem for unconstrained 
hyperelastic image registration that includes all commonly used distance measures. 
It is further shown that solutions satisfy det Vy > 0, which is necessary in many 
applications. The theory is complicated due to the non-convexity of the determinant 
mapping as has been pointed out by Ciarlet [18, p.138f]. Using his notation F = 
Vu: “The lack of convexity of the stored energy function with respect to the variable 
F is the root of a major difficulty in the mathematical analysis of the associated 
minimization problem.” 

This section organizes as follows: A sketch of the existence proof is given in 
section “Sketch of an Existence Proof”. Formal definitions of involved function 
spaces, the set of admissible transformations, and the hyperelastic regularization 
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functional SY? are given in section “Set of Admissible Transformations”. Finally, 
an existence result is shown in section “Existence Result for Unconstrained Image 
Registration’. 


Sketch of an Existence Proof 


This section gives an overview of the existence theory of hyperelastic image 
registration problems. The goal is to outline the major steps, give intuition about 
their difficulties, and introduce the notation. The presentation is kept as informal as 
possible. The mentioned definitions, proofs, and theorems from variational calculus 
can be found, for instance, in [25, Ch. 8] or [18, 20]. 

The objective functional in (1) can also be written as an integral over a function 
f:2xR xR? SR: 


Jit= i f (x. y(&), V(x) dx, (11) 


where the dependence on the template and reference images is hidden for brevity. 

In the following it is assumed that f is continuously differentiable and mea- 
surable in x. This is granted for the part related to the hyperelastic regularization 
functional. For the part related to the distance functional, this can be achieved by 
using a continuous image model 7 and 7 and careful design of D. 

Typically, 7 is bounded below by a constant m := inf, 7[y]. The goal is to find 
a minimizer in a certain set of admissible transformations A, which is a subset of 
a suitable Banach space X. Since X must be complete under Lebesgue norms, our 
generic choice is a Sobolev space W':? (Q,R°), with p > 1, rather than C'(2, R°). 

Existence of minimizers is shown in three steps. Firstly, a minimizing sequence 
{yen C A is constructed, that is, a sequence with limp Ty") =m 
containing a convergent subsequence. Secondly, it is shown that its limit y* is a 
minimizer of 7, in other words, 7[y*] = m. Finally, it is verified that y* actually 
belongs to the admissible set A. 

Key ingredients of the existence proof are coercivity and lower semicontinuity. 

In a finite dimensional setting, the coercivity inequality 


Jly] = C |ly|lx + K for all y € A and constants C > O and K € R, (12) 


guaranties that {y"}, lies in a bounded and closed set. From the lower semicontinu- 
ity it follows 


ya P m= lin Jy leIy 2m, 
00 


and thus y* is a minimizer. However, W'-? (2, R?) is infinite dimensional. Hence, 
bounded and closed sets are in general not compact and the existence of a 
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norm-convergent subsequence cannot be assumed. Thus, less strict definitions of 
convergence have to be used. 

Note that W!-?(92,R*) is reflexive for p > 1. For these cases, it can be shown 
that there exists a subsequence that converges in the weak topology, 1.e., with respect 
to all continuous, linear functionals on W!? (92, R3). Consequently, if 7 fulfills a 
coercivity condition (12), a bounded minimizing sequence can be constructed which 
yields weakly converging subsequence. 

The second part, namely showing that the weakly convergent sequence actually 
converges to a minimizer, is in general more involved. Lower continuity of 7 
with respect to weak convergence typically requires convexity arguments. In many 
examples of variational calculus, the integrand f of 7 is convex in Vy. Thus, the 
integrand f can be bounded below by a linearization in the last argument around 
Vy. Using the weak convergence of Vy, — V y* and noting that y, > y* strongly 
in L* by a compact embedding, the linear term vanishes and lower semicontinuity 
follows; see Theorem 2. 

When using hyperelastic regularization or mass-preserving distance functionals, 
J also depends on the Jacobian determinant det Vy. Thus, the dependence of f 
on Vy is non-convex and further arguments are required to obtain lower semi- 
continuity. To overcome this complication, one can follow the strategy suggested 
by Ball [4]. The idea is to introduce a splitting in (y,cofVy, det Vy) and show 
convergence of the sequence 


{(y*, cofVy*, det Vy")}, C W!?(2,R?) x L4(2, R°*?) x L™(Q,R) =: X 


where the exponents g > 0 andr > O are appropriately chosen. A coercivity 
inequality in the ||- ||y-norm yields a weakly converging subsequence as outlined 
above. Extending f by additional arguments, it follows 


g:2xR? x R*? x R*? x [0,00) > R 
with g(x, y, Vy, cofVy, det Vy) := f(x, y, Vy) 


is measurable in x and continuously differentiable. Most importantly, g is convex in 
the last three arguments. Consequently, weak lower semicontinuity of 7 in X can 
be shown, i.e., (y*, cofVy*, det Vy) — (y*, H*, v*) implies 


jim [eG y's Vy! cotvy'. det Vy") dx = ff g(r." Vy" Hy") dx =m, 
> oo Q 2 


To undo the splitting, the identifications H* = cofVy* and v* = det Vy* must 
be shown. This is uncritical if y is sufficiently measurable, i.e., p is greater than 
the space dimension d. Note that p is also the degree of the polynomial det Vy. 
For hyperelastic regularization, it is possible to show existence under even weaker 
assumptions, i.e., ford = 3 and p = 2. Key is the identity 
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det Vy-I = Vy 'cofVy. 


It will be shown that H* = cofVy* and v* = det Vy* for exponents gq = 4 and 
r = 2; see Theorem 4. 

A final note is related to the positivity of the determinant. To assure that the 
Jacobian determinant of a solution is strictly positive almost everywhere, sufficient 
growth of J is required, i.e., J[y] > oo for det Vy > OF. Note, that det Vy* > 0 
is not obvious. For example, k~! — 0 although k~! > 0 for all k. 


Set of Admissible Transformations 


As pointed out, regularization functionals involving det V y considerably complicate 
existence theory due to the non-convexity of the determinant mapping. Thus, a 
careful choice of function spaces and the set of admissible transformations is 
important for the existence theory to be investigated in the next section. The goal of 
this section is to define the function spaces and the set of admissible transformations 
underlying the hyperelastic regularization functional (10). 

To begin with, the problems arising from the non-convexity of the determinant 
are addressed. One technique to bypass this issue is a splitting into the transforma- 
tion, cofactor and determinant and study existence of minimizing sequences in a 
product space. Here the product space 


X := W!"(Q,R°) x L4(2, R°**?) x L?(2,R) (13) 
with the norm 


I|(y, cofV y, det Vy) |Lx := [ly llw12ce,R3) + lleofV yl 24¢2,R3x3) + || det Vylz2(2,R) 

(14) 
is considered. The space X is reflexive and the norms are motivated by the fact that 
the penalty functions ¢, and w presented in section “Hyperelastic Regularization 
Functionals” are essentially polynomials of degree four and two, respectively. The 
splitting is well defined for transformations in the set 


Ao := {y: (y, cofVy, det Vy) € X, det Vy > Oae.}. (15) 


To satisfy a coercivity inequality, boundedness of the overall transformation is 
required. In the standard theory of elasticity, this is typically achieved by imposing 
boundary conditions; see [4, 18]. In the image registration literature, a number of 
conditions, for instance, Dirichlet, Neumann, or sliding boundary conditions were 
used; see [53]. 

Since finding boundary conditions that are meaningful for the application in mind 
often is difficult, the existence proof in section “Existence Result for Unconstrained 
Image Registration” is based on the following arguments. Recall that images are 
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compactly supported in a domain §2, which can be bounded by a constant M € R. 
Since images vanish outside (2, reasonable displacements are also bounded by a 
constant only depending on {2; see also discussions in [14,58], 


IIvlloo = M + diam(2). 


For larger displacements, the distance functional is constant in y as the 
template image vanishes and thus no further reduction of the distance func- 
tional can be achieved. In this case, the regularization functional drives the 
optimization to a local minimum. Working with the supremum of y, however, 
complicates our analysis as the transformations in the non-reflexive space 
L°. Therefore, an averaged version of a boundedness condition can be 
used: 


satay | fg ¥(a)dx| < M + diam(2). 


This leads to the following characterization of admissible transformations and 
definition of the hyperelastic regularization energy. 


Definition 3 (Hyperelastic Regularization Functional). Let 2 C R?* be a 
domain bounded by a constant M > 0, a € C!(92,Rt) be regularization 


parameters with a@;(x) > a; > O for all x andi = /,a,v, and Ap as in (15). 
The set of admissible transformations is 


A={yeAo : | fo y()dx| < vol(Q)(M + diam(2))}. 


Then the hyperelastic regularization functional is defined as 


Shyper : A > Rt, ShyPerl y] = sesh y] he Sry] 4 Sly], (16) 
with 

Serehly] = fio ae(x) |[Vu(x) dx, (17) 

Sly] = fo Ma(x) e(cofV y(x))dx, (18) 

Sy] = fg v(x) W(det Vy(x))dx, (19) 


and the convex functions ¢ : R°*? > Rt andy: R— Rt, 


2 
oo) = Die max {jn C2 1,0} and y(v) = 2", 
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Existence Result for Unconstrained Image Registration 


This section shows that hyperelastic regularization is sufficient to guarantee the 
existence of optimal transformations for unconstrained image registration problems 
and that the Jacobian determinants of minimizers of 7 in (2) are positive almost 
everywhere. 

There are of numerous ways of showing existence. One option is to follow John 
Ball’s proof for hyperelastic materials [4, 18, 20]. In [14], existence is obtained by 
verifying that the modified hyperelastic regularization functional still satisfies the 
assumptions of Ball’s theorem which yields a very brief and concise result. 

In this section, a new and different approach to the existence theory is given. Its 
main purpose is to give insight into the machinery behind Ball’s proof and to prove 
existence in a simplified setting using only moderate theoretical tools. For the sake 
of clarity, some technical difficulties are avoided by making additional assumptions 
that are no limitation for most applications and also discussed below: 


Al: The distance functional is of the form 


Diy] = fo gp(x, v(x), cofVy(x), det Vy(x)) dx, 


where gp : 2 x R? x R**? x R > R? is continuously differentiable, convex 
in its last argument, and measurable in x. 

A2: The regularization parameters satisfy ag = a, = ay = 1. 

A3: With Id(x) = x for all x, it holds 7 (Id) < oo. 

A4: The boundary 02 of the domain 2 is in C!. 


(A1) holds for a large class of distance functionals, including the ones in [55], the 
mass-preserving distance functional (7), and distance functionals in the Lagrangian 
frame. Differentiability of gp requires a differentiable image model and is assumed 
for simplicity. It can be weakened to the concept of Carathéodory functions; 
see [14,20]. (A2) is made for ease of presentation. In general, there exists a constant 
a > 0 such that for all three penalty functions a;(x) > a. Hence, (A2) present 
no loss of generality. For practical applications, (A3) is satisfied as the distance 
functional is finite for the initial images. (A4) allows to use a generalized Poincaré 
inequality [25, Sect. 5.8] and an integration by parts formula [25, Sect. C.1]. 

The main goal of this section is to prove the following theorem. 


Theorem 1 (Existence of Optimal Transformations). Given are images T,R € 
Img(2), a distance functional D, S"°* and A as in Definition 3. Let the 
assumptions (A1l)-(A4) be satisfied. There exists at least one minimizer y* € A 
of J =D 4 Shyper, 


The proof is divided into a number of Lemmata following the sketch presented 
in section “Sketch of an Existence Proof”. To begin with, coercivity of the objective 
functional in the space X as in (13) is shown. This means that 7 grows sufficiently 
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fast with respect to the ||- || norm; see (14). This will be essential to bound the 
norm of the minimizing sequence. 


Lemma 1 (Coercivity of 7). Under the assumptions of Theorem 1, the functional 
J satisfies a coercivity inequality: IC > 0, KE RVyeEA: 


Tly] = K + Cillyliiie + lleofV yl + || det Vyll72). 


Proof. Let y € A be an admissible transformation. Since D > 0 by assumption, 
coercivity of 7 follows from the coercivity of the regularization term 


Tly] = S™"Ly] = fo IV — Ili. + Pe(cofVy) + (det Vy) dx. 


The penalties ¢, and w are designed such that there exist constants C; > O and 
K, € R such that 


Ib] = khi+C (fo Vy —12, + \|cofVy lz. + (det Vy)) dx. 


Fro 


Using that (a — b)? > 5a° — b* for all a,b € R, it follows that 
Ty] = Ki + C1 (fg FIV I llc = WHllfo + WleofVy life + (det Vy)?) dx. 
With constants C7 > 0 and K> € R, one obtains the inequality 
Tly] = Ko + C2 (IV yllj2 + lloofVy IZ + || det Vyll72) - (20) 


It remains to be shown that the norm of Vy in L* bounds the norm of the 
transformation y. To this end, assumption (A4) is used and a generalized Poincaré 
inequality [25, Sect.5.8] is applied. In addition, the boundedness of the mean of 
transformations in A is used: 


Vyl?.>Gly—-h ax|_ = Csly|?.—|2)(4 aay 
IVylits = Cs\]y — py Jo y@dz|_, = Callylts — 121 (yl Jo yea 


2 
> Callylli2 - 121 (Gy Je ly@lax) 


Due to the assumption on A, there exists a constant K3 € R independent on y such 
that 


IVyllj2 = Ks + Cally lize. (21) 


Combining (20) and (21) and introducing cosmetics constants C > 0 and K € R 
one obtains the desired growth condition 


Tly] = K + C(lly liz + lleofVyllzs + || det Vy 72). 
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Since y was chosen arbitrarily, the assertion follows. 


The following theorem establishes lower semicontinuity of the objective func- 
tional 7. It is a modified version of Theorem 1| of Evans [25, Sec. 8.2]. In contrast to 
the version of Evans, the next theorem does not necessarily assume that ¢ is replaced 
by Vy. However, it is assumed that f > 0, y* > y* strongly, and ¢* — ¢* weakly. 


Theorem 2 (A General Lower Semicontinuity Result). Let 2 C R°? be a 
domain. Let f : 2 x R? x RX + [0,00] be continuously differentiable and 
FT(-,¥,6) be measurable for every fixed y and ¢. Let f(x,y, -) be convex. For 
two sequences y* — y* in L?(Q,R°) with p > 1 and c* — €* in L4(Q,R) 
with q = 1, it holds 


lim inf fy f(x, y*(x), 6(x))dx = foo SOx." (0), OC) dx. 


Proof. The proof basically follows the one of Theorem 1 [25, Sec. 8.2]. Simplifica- 
tions arise from the fact that the sequences { y*}; and {¢*}, converge by assumption 
and f > 0. Another difference is that the second argument of f is used for a 
general sequence C* and its limit C* instead of Vy* and Vy*, respectively. The 
proof is organized in three steps. 


Step 1: Since 0 < f, it follows 
0<m:= lim inf fo f(x,y" (x), OO) dx, 
—>0o 
thus there exists a (sub)-sequence { y*, ¢*}, with 


0<m= im to Fe y* (x), FF) dx. 


Step 2: The strong convergence y -> y* in L? implies convergence almost 
everywhere. Thus, Egoroff’s Theorem [25, Sect.E.2] can be used, which 
implies that for each € > 0 there exists a set 2! C Q@ with vol(2 \ 2!) < € 
such that 


y* —» y* uniformly on 2). (22) 


Next, 92! is reduced by regions where y* or ¢* are too large. Consider 2, := 
QIN{x ER: |y*(x)| + [E*(x)| < eT}. It holds that 


vol(2 \ 2.) + 0 fore — 0. (23) 


Step 3: From f > O and the convexity of f in the last argument (see also [25, 
Sect. B.1]), it follows for each fixed k that 
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So fy b)dx > fo. fx. yo dx 


> fo, FOV OY dx + fo def YOY GO — OY dx. 
(24) 


Due to the uniform convergence of { y*};, see (22) and (23), it follows 
limg+oo fo, £(%, 9" 8") = fo F(x, ¥*, 8%). (25) 


Since also dz f (x, ye, €*) > de f(x, y*,€*) uniformly on Q, and ¢* — &* it 
holds that 


dim fo, de $V OEE — O dx = 0. (26) 
Combining (25), (26), and (24), it follows 
m = limg+oo fo f(x, y*, b)dx > fo f(x,y", H) dx. 
This inequality holds for all « > 0. Using f > 0, the measurability of f, the 


Monotone Convergence Theorem [25, Sect. E.3] and letting « — 0, it follows 
that 


m> fo f(x. y*,&*)dx, 
which concludes the proof. 


Having shown the above general lower semicontinuity result, it is now verified 
that the objective functional 7 satisfies the assumptions that have been made. 


Lemma 2 (Weak Lower Semicontinuity of 7). Given are images T,R € 
Img(2), a distance functional D, S™?*, and A as in Definition 3. Let the 
assumptions (A1)-(A4) be satisfied. 


° There exists a function g : Q x R? x R**? x R3*3 x (0, oo] > R with 


Jly] = / g(x, y, Vy, cofVy, det Vy) dx 
2 


that is continuously differentiable, convex in its last three arguments, and 
measurable in its first argument. 
¢ The objective functional J is weakly lower semicontinuous, i.e., 


timint [ g(x, y*, Vy", cofVy*, det Vy*) ax | g(x,y, Vy, H,v) dx 
c—>O0O Q Q 
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whenever y* — y inW!?, cofVy* = HinL*, detVy* — vin L?. 


Proof. Both results are proved separately. 
First, 7 is rewritten as 


TI = [ go(x.y.Vy.cotVy.detVy)) dx 
2 


" i: a(x) |Vy[? + ag(x) 6(cofVy) + ay(x) W(det Vy) dx 
2 


and combine the integrands to g : 2 x R® x RB? x BR? x (0,~] > R. 
By assumption on D and design of S*Y?*, the mapping g(x, y, -, -, +) is convex 
for fixed x and y. Furthermore, since the images and the functions ¢ and w are 
continuously differentiable, the integrand g is continuously differentiable. Since the 
images are measurable, g is measurable in x for fixed (y, D, H,v) € R? x RS x 
R3*3 x (0, oo]. 

For the second part, one notes that the weak convergence of y* to y in W!? 
implies strong convergence of y* in L? due to the compact embedding of W!* CC 
L?; see [25, Sect. 5.7]. Hence using the first part yields weak lower semicontinuity 
of J on X by applying Theorem 2. 


Due to the coercivity of 7 shown in Lemma 1, there exists a bounded minimizing 
sequence {y*, cofVy*, det Vy"; C X, where X is the product space as in (13). 
Lemma 2 shows lower semicontinuity of 7, thus, the sequence converges to a 
minimizer (y, H,v) in X. In the next step, the relation between the components 
y, H and vis established. More specifically, a continuity results for the cofactor and 
determinant mappings for function in W!? (2, R°) is given. 


Theorem 3 (Weak Continuity of Determinants). Assume d < q < co and y* = 
y in W!4(Q, IR). Then det V yk — det Vy in L4/4(2,R). 


Proof. See Evans [25, p. 454] for a complete proof. 


This result is expected as the minimizing sequences {y*}, are bounded in the 
spaces W!7(2, IR“) and {det Vy“ }; as a polynomial of degree d can be bounded 
in L“(2,R). Combining these two bounds yields the weak convergence of the 
sequence {det Vy}, in L4/4(Q, R). This result is used in [58] to show existence of 
solutions to volume constrained image registration in W!-? and actually extended to 
pad. 

Using Theorem 3, one option to obtain the identification v = det Vy in L? is to 
tighten the measurability demands on y by, e.g., resorting to W!°(, R*) similar to 
[60]. 
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In hyperelastic image registration also the cofactor matrix is regularized by the 
area regularization functional. This can be used to improve the above result; see 
also [4, 18]. This allows to achieve an improved continuity result based on [18]. 


Theorem 4 (Weak Continuity of Cofactors and Determinants). Let Q C R° 
fulfill (A4). Then it holds 


7 —y in W!?(Q2,R°), 
cofV y* — H in L4(@,R°%3), } > 
detVy§ sy in L2(2,R) 


H =cofVy, 
v=detVy. 


Proof. This proof combines the proofs of Theorems 7.5-1 and 7.6-1 in [18] and is 
structured into two steps. 
For ease of presentation, define I := cofVy. 


Step 1: Weak continuity of the cofactor mapping. Without loss of generality, 
presentation is focussed on the top-left entry of the cofactor matrix, i.e., to 
show that I\;(x) = A(x) for almost every x € 2. The result extends 
straightforwardly to the remaining entries. 

To begin with, it is assumed that y € C?(2, R*). Then it holds 


I, = 9220393 — 02303 y2 = 03(V302y2) — 02(V3032). 


Since y is continuously differentiable and 0§2 is smooth one can integrate by 
parts, see [25, Sect. C.2]. Thus for test functions ¢ € C§°(2, R) it holds 


Jou 6 dx = — fa 3022036 dx + fi y303y2026 dx. (27) 


This equality is now extended to functions y € W!?(, R*) by the following 
approximation argument. Since C~(92,IR*) is dense in W!?(2,R*) [25, 
Sect. 5.2], one can approximate y by a sequence of functions in C? for 
which (27) holds. Note that for ¢ being fixed both sides of (27) are continuous 
in the space C? with the norm || - || 1.2, since 


lhe M6 dx| < Walle Willen < A® bala (28) 


and fori, j,/ = 1,2, 3, 


Mo v9) 181 dx| < yell; yelles Eller < CoG) ly lhyr2- (29) 


Hence, (27) holds also for y € W!*(,R?) and thus its right hand side can 
be used to prove the desired equality /) = H. To this end, consider a fixed 
test function ¢. By Hélder’s inequality [25, Sect. B.2], it holds that the bilinear 


mapping 
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B(-,-): L°(@,R) x W!?(2,R) > Rwith (fg) fo f 9:8 Ino dx 


is continuous. Due to the compact embedding W!2 co L? [25, Sect. 5.1], it 
holds 


yk = yin W!7(Q,R3) = y* > y in L?(2,R’). 


Because of the continuity of the bilinear form B, the strong convergence of 
f{y"},, and the weak convergence of {0; vy he, Theorem 7.1-5 in [18, p.348f] 
can be used to obtain that for alli, j,/,m € {1, 2, 3} 


y* — yin W'7(Q,R°) > fo yF A; yf Omg dx > fo ¥i A; ¥1 Imo ax, 


which due to (27) implies [,(cofVy*)i1édx > {,(cofVy)11¢dx. Note that 
H, is by assumption the weak limit of {(cofV y*),;}, in L*+(Q@, R). Therefore, 
(cofV y)11(x) = Ay,(x) for almost every x and thus both functions are equal 
in L4(Q2,R). Repeating these arguments for the remaining matrix entries yields 
cofVy = H in L4(2, R**?). 

Step 2: Weak continuity of the determinant mapping. It is now shown that under the 
above assumptions det V y(x) = v(x) for almost every x € 92. 
Recall that (det A). = A(cofA)' and hence detA = eS ay; (cofA),;. 
Starting with y € C?(2, R?), 


det Vy = )o@jy)- Ty = >) abi) (30) 


j=l j=l 


where the last equality holds, since the cofactor is divergence free [25, 
Sect. 8.1]: ae 0; Tj, = Oforalli = 1,2,3. Since y is continuously 
differentiable and 02 is smooth, one can integrate by parts [25, Sect.C.2] and 
see from (30) that for all test functions € € C§°(2, R) 


3 3 n 
[Lei nitax= f Vabarltds=—f Yon ty d;§ dx. 


j=l j=l 
(31) 


" _, 11; 0;6 dx is continuous with respect 


j=l 
to the W!?(2,R*) norm. By applying the same approximation argument as 
above, one sees that functions y € W'!7(2,R°) are divergence free in the 


following sense: 


For fixed ¢, the mapping y +> /[, @a 


/ SY (cofVy)1; 0; dx = 0 for all £ € CH°(2,R). (32) 
2 j=] 
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Now (31) is extended to functions y € {y € W!7(Q,R*) : cofVy ¢€ 
L4(@, R**?)} by showing that for all test functions ¢ € CS°(2, R), 


3 3 
/ Lens sax =- f yon I; 0;6dx. (33) 
2 j=] 2 j=] 


To this end, note that for fixed ¢, both sides of this equation are bounded bilinear 
forms in y; and I and thus continuous. Therefore, it can be assumed that y; € 
C™(2,R*). But then it follows that (y;0) € C§°(2, IR) and thus 


3 3 
er rjajdx+ f YS jy) Li; ¢ dx, 
2 


j=l j=l 


3 
0= f AO de = | 


which yields the desired extension of (31). 
Given that y* — y in W!7(Q,R°) and cofVy* — cofVy in L4(2,R**°), 
one obtains the continuity result 


3 
: det Vy* ¢ dx = -| Y— yf CofV y*)1; 9; dx 
2 


j=l 


+-f Y nleotVy);ajtax = f det Vy € dx 
2 Q 


j=l 


as in the first part of the proof from the compact embedding of W!?(2, R°) 
into L?(2,R°*). Note that for fixed ¢ both sides of (33) are bounded bilinear 
functions. 

From det Vy* — v and detVy* — detVy in L?(Q,R), it follows that 
v = det Vy almost everywhere, which concludes the proof. 


This completes the ingredients for the proof of the main result in Theorem 1. 


Proof (Proof of Theorem 1). By assumption, 7 is bounded from below. Thus, there 
exists a minimizing sequence {y*}, such that 


lim J[y*] = inf Z[y*] =: m. 
k->00 yeA 


Note that 7 is finite for Id(x) = x. Therefore, it can be assumed that {7[y*] : k € 
N} is also bounded from above by a constant M > 0. 

By Lemma 1, the sequence {(y*, cofVy*, det Vy")}; can be bounded in the 
Banach space X defined in (13) in terms of M. More precisely, there are constants 
C > Oand K € R such that for all & it holds 
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M > I[y*] = K+ Cllly* ipa + lleofVy* Ibs + || det Vy* [[32). 


Since X is a reflexive space, there exists a subsequence — again denoted by 
{(y*, cofVy*, det Vy*)}, — that converges weakly to a (y, H,v) € X. 
Lemma 2 proofs 7 to be weak lower continuous. Thus 


timint [ g(x, y*, Vy*, cofVy*, det Vy") dx > / g(x,y, Vy, H,v) dx. 
k->oo Jo Q 
Theorem 4 yields the identifications H = cofVy and v = det Vy. 

The last step is to show that y is admissible. It remains to show that det Vy > 0 
almost everywhere. To this end, consider a fixed € € [0, 1) and the set 


Sei= {xe 2 : det Vy(x) < €}. 


It then holds that 
W(e) vol(S.) < / Ove / det yd 
Se Se 
<|/ g(x,y, Vy, cofVy, det Vy) dx 
Se 


< timint [ g(x, y*, Vy", cofVy*, det Vy*) dx < M. 
oo Se 


Due to the growth condition w(v) > oo for vy > 0°, it follows that So has zero 
volume. Thus, det Vy > 0 almost everywhere in S2 and y € A, which completes 
the proof. 


To summarize, the hyperelastic regularization functional SYP guarantees the 
existence of optimal transformations for the unconstrained variational image reg- 
istration problem. The proof of Theorem | illustrates difficulties due to the 
non-convexity of the objective functional and shows remedies. The main idea 
is to split y into Jacobian, cofactor, and determinant components and to show 
convergence of a minimizing sequence in a product space by proving coercivity and 
lower semicontinuity; see Lemmas | and 2. Subsequently, weak continuity of the 
cofactor and determinant mapping is used to undo this splitting; see Theorem 4. The 
provided proof is a special case of the existence theory of polyconvex functionals; 
see [4, 18, 20,25]. In addition to providing existence of optimal transformations, the 
theory shows that a solution y satisfies det Vy > 0 almost everywhere. 

Instead of striving for the most generalized result, this section intends to provide 
insight into machinery of [4]. Therefore, four simplifying assumptions were made 
that are, however, uncritical in most practical applications. 
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4 Numerical Methods for Hyperelastic Image Registration 


Generally, image registration problems cannot be solved analytically. Thus, 
numerical methods are used to approximate an optimal transformation. This 
section presents numerical methods for hyperelastic image registration based on 
a discretize-then-optimize approach. The first step is to properly discretize the 
variational problem (2). Typically, a coarse-to-fine sequence of finite dimensional 
optimization problems is generated, where all problems are linked by the 
underlying variational problem (2). Each problem can then be solved using 
standard optimization methods. Note that the coarse-to-fine strategy adds further 
regularization and additionally speeds up the computations. 

In the hyperelastic setting, discretization is not straightforward. A challenge 
is to ensure one-to-one numerical solutions to the optimization problems, which 
are mandatory in most practical applications and guaranteed in the continuous 
function space setting. It is well known that a discrete object does not necessarily 
mimic all the properties of its continuous counterpart. For example, a function that 
is non-negative on a discrete set may not be non-negative everywhere. Thus, an 
approximated solution may not satisfy det Vy > 0 almost everywhere, although the 
above existence theory guarantees this for the solutions of the continuous problem. 

This section organizes as follows. Section “Discretizing the Determinant of 
the Jacobian” illustrates difficulties arising in the discretization of Jacobian deter- 
minants using finite difference schemes. As one way to avoid these difficulties, 
Galerkin finite element methods are presented in section “Galerkin Finite Element 
Discretization”. The main idea is to compute a solution to the variational problem 
in the space of globally continuous and piecewise linear transformations. A key 
benefit is that for these transformations geometric quantities like area and volume 
can be computed exactly; see section “Galerkin Finite Element Discretization”. 
Consequently, solutions of the discrete problem are guaranteed to be one-to-one. 
Finally, section “Multi-level Optimization Strategy” presents multi-level numerical 
optimization approaches that solve the finite dimensional optimization problems; 
see also [55]. 


Discretizing the Determinant of the Jacobian 


The variational image registration problem (2) can be discretized using finite 
difference schemes. The underlying idea is to approximate the solution to the 
variational problem on a grid, which is essentially a collection of points. Many 
commonly used algorithms for image registration are based on these principles; see, 
e.g., [55] for an overview. 

However, deriving finite difference discretization of operators such as the 
determinant of the Jacobian is not straightforward as illustrated by a simplified 2D 
example. For a transformation y € C?(2, R?), the determinant of the Jacobian is 
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det Vy(x) = 9191 (x) 02y2(x) — d2y1(x) O1y2(x). (34) 


Note that all entries of Vy(x) are coupled, which causes difficulties for finite 
difference discretizations. This section analyzes this problem in detail and presents 
a finite volume technique that can be used to obtain a proper discretization. 


Finite Differences in 1D 
To start with the analysis, consider in a 1D setting discretization of the energy 


Sly] = : [ (ay(x))°dx, (35) 


where §2 = [0,1] is divided into m cells of width h = 1/m. The cell-centered 
grid x° € R”™ and the nodal grid x" € Rt! are defined by xf = (i — 0.5)h and 
x!" = (i — I)h, respectively; see [55] for details. Assuming that y € C?(2,R) and 
a short central finite difference yields 


yxy) — y(x?) 


2 
; + O(h*). (36) 


dy(x;) = 


Note that the discretization is of second order for a price of a grid change. The 
function y is approximated on the nodal grid, whereas its derivative is approximated 
on the cell-centered grid. With the discrete partial differential operator ue 
R@t+1 > R" 


gr = te My, € Ree. (37) 
-11 


Equation (36) reads dy(x°) = 0", y(x") + O(h’). To avoid the grid change, one 
could also use the long stencil dy(xf) © (y(x},,) — y(xj_,))/(2h). The price to be 
paid is the non-trivial null-space with high frequency components which are unseen 
by this discrete regularizer. While this might be tolerable on a fixed discretization 
level, it is a crucial drawback for multigrid schemes. Therefore, this discretization is 
not recommended. Finally, one could also think about a one-sided stencil, dy(x/) ~ 
(y(x},,) — y(x;))/h. Here, the price to be paid is that this approximation is only 
first order and this discretization is also not recommended. Thus, 0”, is preferable for 
approximating derivatives of nodal quantities. Derivatives of cell-centered quantities 
can be approximated analogously on a subset of the nodal grid x” using ae 

Using a nodal discretization y” ~ y with y’? % y(x!) and the above 
discretization of the differential operator, a second-order approximation of (35) is 
obtained by using a midpoint quadrature rule: 
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Fig. 4 Overview of grids and indexing convention in 2D as also used in [55]. A cell-centered grid 
x a nodal grid x"and staggered grids x*! and x*? are depicted in (a). The indexing convention 
for the (i, 7 )th cell is illustrated in (b) 


h 
Sly") = 50") Ga), y". 


Note that both the finite difference operator and the quadrature rule are of second 
order which is essential for a consistent approximation. 


Finite Differences in 2D 

Already in 2D, the situation gets far more interesting. Let the rectangular domain 
2 = [0, 1]? be divided into (m, m) cells of uniform edge length h = 1/m. Similar 
as above, cell-centered and nodal grids are defined as 


See Fig. 4 for an overview of grids and indexing conventions. For ease of presen- 
tation of finite difference operators to higher dimensions, the concept of Kronecker 
products is convenient; see also [9]. 


Definition 4 (Kronecker Product). Given two matrices A € R?!*?? and B € 
R¢'*@, the Kronecker product is defined by 


a1i1B Fens a1,p,B 
A®B= : a ; E RMUX P22 | 


Ap, 1B... Ap, ,p)B 


Non-linear Image Registration 2035 


As the following discussion illustrates, it is impossible to give a second order 
approximation to all components of the Jacobian matrix of y = [yi,y2] € 
C?(@,R*) on a cell-centered grid by using short differences. To illustrate this, 
consider the first component y, of the transformation. Second order approximations 
to the partial derivatives of a nodal discretization y? € R™+)” can be obtained 
using the finite difference operators 


gh — gh Q Tas E Rent lx (n+ 1)? 
m . 


m,1 


= In+1 ® at and ah 


m m,2 


In contrast to the 1D case, partial derivatives are not approximated on a cell-centered 
grid, but on staggered grids. Here, the first partial derivatives 0” jf” are 


h 
: : : med and On 
approximated at different locations 


see also Fig.4. As in the 1D case, grid changes are inevitable when using 
short differences. Furthermore, different partial derivatives are approximated at 
different positions. This also holds for other choices of discretization. For example, 
discretizing y, on the staggered grid x*! and using the operators 


h h h h 
On = In ® On and On2 = i a ® In+1, 


approximates 0, y; on a cell-centered grid and 02y; on a subset of a nodal grid. 
Note that for all combinations of the above grids and finite difference operators the 
components of the gradient of y; are approximated at different locations. This is 
a major source of difficulty when discretizing regularization functionals based on 
coupled partial differential operators such as the determinant of the Jacobian (34). 


Finite Volume Discretization 

The above presentation indicates that second order approximations of the partial 
derivatives at the cell-centered grid with short differences are impossible. Averaging 
derivatives to the cell-centers is not a remedy as it annihilates oscillatory functions. 
One way to overcome this limitation is to approximate det Vy in a finite volume 
approach by measuring volume changes as it has been proposed [36]. In [36], the 
volume of a deformed cell V is approximated by the sum over of the volume of two 
triangles spanned by its vertices V;, see also Fig. 5: 


i det Vy(x)dx = dx 
V VV) (38) 


=(V3 — Vo) x (V2 — Vi)/2 + (Va — Vi) x (V3 — Va) /2 + OC’). 
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Fig. 5 Measuring the volume of a transformed cell as in [36]. The volume of the transformed 
cell is approximated by the sum of the volumes of the two triangles spanned by its vertices; see 
(a). In (b), a non-invertible transformation is applied to the cell. Note that although the volume of 
the triangle spanned by V4, V2 and V' is negative, the volume of V is positive. This motivates to 
individually control volume changes of the triangles in order to detect twists 


Assuming y € C?(92,R°), this approximation is second order; see also section 
“Images and Transformation’. The first equality requires that y is invertible due to 
the transformation theorem, which is to be ensured by hyperelastic regularization. 
An advantage of this approach is that transformations that are not one-to-one can be 
detected. Note that for a non-invertible transformation and a sufficiently fine spacing 
of control points there is a sign change of the volume of a triangle; see Fig. 5. This 
approach was also used for hyperelastic image registration in [14]. 

Note that using this discretization, the constraint det Vy > 0 is controlled only 
approximately and on a tetrahedral partition rather than point-wise. In order to 
ensure det Vy > 0 almost everywhere, additional assumptions on the behavior of 
y between the control points are required. In [14] it is shown that based on the 
above discretization the volume constraint can be ensured almost everywhere for 
continuous and piecewise linear transformations. Motivated by this observation, the 
next section presents a Galerkin Finite Element approach that operates on this finite 
dimensional space. 


Galerkin Finite Element Discretization 


This section presents a Galerkin finite element method based on the general 
framework presented in [32]. The idea is to discretize the variational problem (2) 
with regularization functional S'Y?* using globally continuous and piecewise linear 
transformations on a tetrahedral mesh. For these functions, the constraint det Vy > 
0 is fulfilled everywhere if it holds on each element. In accordance with the theory, 
our discrete solutions will share this important quality. 
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Let Vj,...,Vny € IR? denote vertices and T;,..., T,, tetrahedra, where it is 
assumed that vol(7;) > O for alli = 1,...,7. This builds a finite dimensional 
subspace 

An := {y €C(2,R’): y\7, € I7'(T, R*) fori = 1,...,n7} C A, (39) 


where fi is related to ny and denotes order of the approximation, [7, denotes 
the space of first order vector-valued polynomials. Standard nodal Lagrange that 
functions b!,...,b"” : 2 — R are used as a basis of Ay. The coefficients y of 
y" € A,with respect to that basis are stored component-wise in a column vector of 
size 3ny, yj = Gp yn. i =1,...,ny: 


y*(x) = ) yb! (a), (40) 


j=! 


where y; = y"(V;) and analogously a discretization of a reference transformation 
Yref 1S Obtained as (Yret); = Yref(Vj). The discretized hyperelastic registration 
problem then reads 


J RY + Rt, Jy) = Dly) + aS™"(y — Yret). (41) 


The discretization of the distance and hyperelastic regularization functional are 
derived as follows. 

Let J, € R*** denote the identity matrix, 1, € R* the vector of all ones, 
® the Kronecker product, and © the Hadamard product. The entries of the Jacobian 
matrix Vy" are piecewise constant on each triangle. The nine components of the 
Jacobi matrix are stored in a column vector (By) € R®”’, where B is the discrete 
vector gradient operator formed essentially from the discrete partial derivatives 
af € R'™*"¥ with (0%); ; = deb! (Vj): 


B=Lh@V", with(V")T = (ap, (atyT, (04)"). (42) 


Interpolation from nodes to barycenters of the tetrahedra is accomplished by 
applying the averaging matrix 


1/4 if V; is node of T; 
A=1@A€e R'™*"v with Aij = / i Vv; 1s node o (43) 
0 otherwise 
Assembling the volumes of the tetrahedra in 
v € R”’, with v; := vol(7;) and setting V := diag(v), (44) 


the SSD distance functional (5) is approximated by using a midpoint quadrature rule 
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D(y) ® Sres(y) ' Vres(y), where res(y) = 7 (Ay) — R(x). (45) 


Since the Jacobian matrix of y’ € .A, is piecewise constant, the hyperelastic 
regularization functional is evaluated exactly by 


SHY) =F(y — Yeo)’ BT (Ip ® V)B (y — Yret) ~ 
+ avid (cofBy) + ayv' y(det By). 


In line with the organization of the gradient, the entries of the cofactor matrix are 
stored in a column-vector of length 9n;. The first derivatives of the discretized 
objective function is 


dJ(y) = dD(y) + dS®™"(y) + dS**(y) + dS(y). (47) 


Using the chain rule yields 


dD(y) =res(y)'V(VT(Ay))A, (48) 
dS*8"(y) =a (y — Yret)' B' (Io @ V)B, (49) 
dS***(y) =a, ((19 @ v) © $'(cofBy))' d cofBy, (50) 
dS“"(y) =a, v' (det By) d det By. (51) 


Setting D/ = diag(d;y/) € R"™*"", one gets 


D3 —D3 —D? D3 
Ds —D3 —D} D? 
D3 —D} —D32 D? 
BD; —D; D, 
dcofBy = D3 —D} —D; D! Be Rrx3ry | 
D;-D} -D} D} 
DZ -DZ = —D}3 D, 
De —D?—D} Dt 
D3—D? Di Di 


. (52) 
With the abbreviation C/ = diag((cofBy);,;) € IR"’*", the derivative of the Jacobi 
determinant can be compactly written as 


adtey=(C).C?.C).C).c7, CC). C,.c) Ber’. (53) 
Following the guidelines of [55], the Hessian of the objective functional is approx- 


imated to ensure positive semi-definiteness and avoid computations of second 
derivatives of the (generally noisy) template image as follows 
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H(y) © doD(y) + doS'™" + doS***(y) + doS”"(y) (54) 


with the summands 


dyD(y) = AT(VT(Ay)) /WVT(Ay)A, 09) 
d)S*8 = oy BT (Ip ® W)B, 28) 
dS***(y) = at,(dcofBy)' (diag(19 ® v) 6” (cofBy)) dcof By, (57) 
d>S*(y) = ay(d det By)" diag(v © w” (det By)) d det By. (68) 


Note that for this distance functional the dependency of the first summand on y is 
only low order and the second summand is constant with respect to y. In contrast, 
the Hessian of the area and volume regularization functionals strongly depend on y. 


Multi-level Optimization Strategy 


Using the above presented Galerkin method translates the variational image regis- 
tration problem into a finite dimensional optimization problem. Solving the discrete 
problem is not straightforward as it is typically large, nonlinear and non-convex. 
However, state-of-the-art tools from numerical optimization can be used to obtain 
accurate and efficient schemes. This section describes a multi-level strategy using 
Gauss—Newton optimization. 


Multi-level Idea 

In image registration, challenges arise from the non-convex dependence of the 
objective functional on the transformation to be determined. Thus, many local 
minima have to be expected; for an example, see [55, p. 112]. A typical solution 
strategy is to approximately solve the variational problem (2) on a coarse-to-fine 
hierarchy of discretizations often referred to as multi-level strategy [55] or cascadic 
multigrid [7]. The key motivation is to reduce the risk of being trapped in a local 
minimum and to obtain good starting guesses for the correction steps on finer 
discretization levels. A positive side effect is a reduction of computational costs. 

Generically, a nested series of finite dimensional spaces Aj, with 


An, C An C+ CA 


is constructed, where h; refers to the approximation order. The idea then is to 
compute a minimizer y! of 7 in Ap, first. This is relatively simple, as it is assumed 
that the space .A;,, is smooth and the problem is of small size. Note that y! is in 
Sp, and can thus be used as starting guess. In practice, only the coefficients of the 
basis functions in S;,, at the new vertices need to be computed. For Lagrange basis 
functions, weights are easily obtained by evaluating y! at the nodes of the mesh 
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Fig. 6 Subdivision used for generating nested families of structured meshes as in [14]. Left: 
triangulation of an m = [2, 2] grid. Right: tetrahedral subdivision of a voxel 


of S;,. With this excellent starting guess, computing a solution in A,, is relatively 
simple, particularly with a second order schemes. This procedure is repeated until 
the finest level is reached. 

There are many ways to construct a nested sequence of triangular meshes. A 
simple but powerful method is to use nested rectangular meshes and decompose 
each cell into tetrahedra as depicted in Fig. 6. However, more efficient strategies 
using local refinements based on error estimates exist; see, e.g., [39]. 

Commonly, the representation of the images are simplified on a coarse level by 
reducing the data; see [55]. Thereby, only essential image features are considered 
and consequently the danger of being trapped in a local minima is reduced further. 


Gauss Newton Optimization 

On each level, a Gauss—Newton scheme [57] can be used to compute a minimizer 
of the discretized objective function J in (41). Until convergence, the objective 
function J(y), its gradient d/(y), and an approximation H(y) to its Hessian are 
computed. The search direction v is then obtained by solving the linear system 


H(y) v= —dJ(y). (59) 
As outlined in section “Galerkin Finite Element Discretization”, a symmetric and 
positive definite approximation of the Hessian is computed. Solving the above linear 


system is one of the most computationally expensive parts of the scheme. Recall (54) 
that the Hessian consists of four summands 


H(y) © dyD(y) + doS™8" + doS***(y) + doS”"(y). 


The Hessian of the length regularizer S'*"2" is simply a discrete Laplacian for 
each component of the transformation and does not depend on y unlike the other 
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Fig. 7 Nonzero patterns of the Hessians of length (left), area (middle), and volume regularization 
functionals (right) for a tetrahedra mesh based on (3,3,3) cells. The mesh has 199 nodes and a 
number of nonzero elements in the Hessians are 5,904, 18,963, and 18,963, respectively 


summands, which need to be updated in each iteration. Recall that a tight coupling 
between the components of the transformation is introduced by the area and 
volume regularizer and that the effective weight of d7S’°! depends on the current 
iterate. A visualization of the nonzero patterns of the four blocks is presented in 
Fig. 7. 

Since H is symmetric and positive semi-definite by design, a conjugate gradient 
(CG) method [44] can be used to approximate v. Conjugate gradient methods 
become even more efficient when being combined with preconditioning; see, 
e.g., [8]. Suppose there is an approximation C € R‘™** to H that is positive definite 
and easily invertible. Given a starting guess vo € IR™ consider the iteration 


vy) = vo —aC | (H[y]vo +d J[y]) 


with some step length wa. Using C = H anda = 1, this would yield the result in 
one iteration. However, it is equivalent to directly solving (59). Thus, the idea is to 
use a “sufficient” approximation of C to H that is considerably easier to invert than 
fH itself. 

There are many preconditioning techniques for conjugate gradient methods. A 
computational attractive option is Jacobi preconditioning, 1.e., C = D, where 
D = diag(#). More expensive but also more effective preconditioning is offered 
by Gauss-Seidel preconditioning, i.e., C = (D + L)D~'(D +L"), where L is 
the lower triangular matrix of H and D the diagonal of H; see [8, Ch.4] for more 
details. 

The step length is determined by a standard Armijo line search in combination 
with a backtracking that ensures det Vy > 0; see [57]. Standard stopping rules based 
on the value of the objective functional, the norm of the gradient, and the norm of 
the update can be used as discussed in detail in [55]. 
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Fig. 8 3D Registration result for diastolic and systolic PET data of a human heart. The first row 
shows volume renderings of reference, template, and transformed template image. The second row 
shows slice projections of the image data and the estimated transformation. Image data provided 
by Fabian Gigengack, European Institute of Molecular Imaging, University of Miinster, Germany 
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5 Applications of Hyperelastic Image Registration 


This section outlines the potential of hyperelastic image registration for real-world 
applications. Exemplarily, two registration problems from medical imaging are 
considered. In both applications mass-preserving distance functionals are required 
and hyperelastic regularization is used to yield mathematically sound formulations. 


Motion Correction of Cardiac PET 


Positron Emission Tomography (PET) can provide useful information about the 
metabolism of the human heart, which is useful for the diagnosis of coronary artery 
diseases. To this end, a radioactive glucose tracer is administered and emission 
events are recorded by a detector ring during its decay. Finally, a tomographic image 
representing the spatial distribution of the tracer is reconstructed. Typically, a large 
fraction of the tracer is metabolized by the left ventricular muscle of the heart, which 
is thus visible in the images; see Fig. 8. 

A key problem in PET reconstruction is the severe degradation of the images 
due to respiratory and cardiac motion during image acquisition. In principle, the 
more emission events are recorded, the better the expected image quality. Thus, 
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acquisition times are typically in the order of minutes and motion is inevitable. 
The so-called gating techniques are typically used to compensate for these motion 
artifacts [21]. The idea is that recorded emission events are grouped into a number of 
gates, which relate to particular phases in the respiratory and cardiac cycle. For each 
gate, a reconstruction is computed which shows less motion blur, but is also based 
on fewer counts and consequently of degraded quality as signal to noise has been 
reduced. To take full advantage of all measurements, the individual reconstructions 
are aligned using image registration and finally fused [30, 61]. 


Mass-Preservation 

PET images represent the distribution of the density of the radioactive tracer, in this 
case, the distribution of a glucose tracer measured 1h after injection. Therefore, it 
can be assumed the amount of tracer per given tissue unit to be constant. Thus the 
mass-preserving distance measure, as defined in section “Distance Functionals”, is 
used 


Diy] =D 17 Ry] = 5 | 7b: det Vy — R)?dx. 
2 Je 


As mentioned in section “Distance Functionals”’, the mass-preserving distance func- 
tional DY? is not convex in Vy and mass-preservation requires that transformations 
satisfy det Vy > 0. Also, large strains for cardiac motion correction are expected 
and images will be of relatively poor quality when using fine gating schemes. 
Thus, hyperelastic regularization is used since Theorem | guarantees existence of 
minimizers with positive Jacobian determinants. 

The mass-preserving distance functional is discretized following the guidelines 
of section “Galerkin Finite Element Discretization”. As for the SSD distance, a 
midpoint quadrature rule is employed and computation of the Jacobian determinant 
and its derivatives are reused. 


Registration Results and Impact on Image Quality 

Registration results for a 3D PET data set of a human heart are visualized in 
Fig. 8. More precisely, Fig. 8 only presents results for the registration of diastolic 
and systolic phases which is the most difficult scenario. Data courtesy by Fabian 
Gigengack, European Institute for Molecular Imaging, University of Miinster, 
Germany. It turns out that the mass-preserving hyperelastic registration approach 
accurately deforms the contracted systolic gate such that is very similar to the 
diastolic gate. The smooth and invertible transformation is visualized using a slice 
projection. 

The image quality that can dramatically improved by registration and subsequent 
averaging of the transformed images as shown in Fig. 9. Here, a dual gating into five 
respiratory gates each being divided into five cardiac gates is performed, yielding 
25 image volumes. Each single gate contains only a small fraction of the signal, 
however, is less affected by motion; cf. Fig. 9a. Reconstruction without registration 
yields a smoother image degraded by motion blurr; cf. Fig.9b. A smooth and 
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single gate without motion correction motion correction result 


Fig. 9 Motion Correction result for PET data of a human heart. Slice projections of a single 
gate (/eft) a blurred reconstruction without motion correction (center) and motion-corrected 
reconstruction (right) are shown. Image data provided by Fabian Gigengack, European Institute 
of Molecular Imaging, University of Miinster, Germany 


motion-corrected reconstruction is obtained by following the pipeline suggested 
in [61] and averaging the aligned images. Here a much sharper and clearer image 
can be reconstructed; cf. Fig. 9c. 


Summary and Further Literature 

Mass-preserving hyperelastic image registration reduces motion artifacts in cardiac 
PET. The transformation model respects the density property of PET. Hyperelastic 
regularization is used to ensure existence of solutions that satisfy the mass- 
preserving constraint. The mass-preserving transformation model was validated 
in [30] and a pipeline for practical motion correction is suggested in [61]. 


Susceptibility Artefact Correction of Echo-Planar MRI 


Echo Planar Imaging (EPI) is a commonly available ultrafast MRI acquisition 
technique; see [66]. It is routinely used for key investigation techniques in modern 
neuroscience such as diffusion tensor imaging (DTI) or functional MRI (f{MRID). 

While offering considerable reduction of acquisition time, a drawback of EPI 
is its high sensitivity against small perturbations of the magnetic field. Field 
inhomogeneities are inevitably caused by different magnetic susceptibilities of soft 
tissue, bone, and air and thus present in any practical setting. Inhomogeneities 
result in geometrical distortions and intensity modulations that complicate the 
interpretation of EPI data and their fusion with anatomical T1- or T2-weighted 
MR images obtained using conventional acquisition techniques with negligible 
distortions. 

This section summarized the tailored variational image registration method for 
the correction of distortions presented in [62]. The method can be seen as a special 
case of mass-preservation hyperelastic registration for transformations that are 
restricted along one a priori known spatial direction. 
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Fig. 10 Illustration of the Reversed Gradient Approach [15]. The initial measurements (first 
column) are distorted in opposite directions, depicted as white arrows. The estimated field 
inhomogeneity B (second column) gives rise to two geometric transformations (third column) that 
can be used to approximately correct the data (right column). Data courtesy by Harald Kugel, 
Department for Radiology, University of Miinster, Germany 


Reversed Gradient Method 

The Reversed Gradient Method was firstly described by Chang and Fitzpatrick in 
1992 [15] to correct for distortions due to field inhomogeneities. They derived a 
physical model for the distortions, which shows that measurements are distorted 
along one a priori known spatial direction. Interestingly, the direction of distortion v 
is not only known, but can also be controlled by parameters settings on the scanner. 
It is thus possible to acquire a pair of images, denoted by Z, and 7, that are 
oppositely affected by distortions; see also Fig. 10. 


The Variational Problem 

The measurements Z; and Z, represent density distributions of protons. Therefore, 
there is a mass-preserving property [15] is assumed. Due to field inhomogeneities, 
the signal is wrongly localized along the distortion direction v and the image 
intensities are modulated depending on the determinant of the Jacobian of the 
transformation (Fig. 11). Chang and Fitzpatrick describe the relation of the distorted 
measurements to the undistorted image Z by 


I(x) = Z)(x + vB(x))(1 + 0,B) = h(x — vB(x)) + 0,B), (60) 
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Fig. 11 Improved correspondence of corrected EPI measurements to anatomical MRI. Data 
courtesy by Harald Kugel, Department for Radiology, University of Miinster, Germany 


where B : $2 — R denotes the field inhomogeneity that needs to be measured 
or estimated numerically. Note that for distortion directions +v, there are two 
transformations y+ = x + vB(x) and hence det Vy = 14 0,B. 

Considerable progress has been made in the numerical treatment of the reversed 
gradient method in the last decade; see, for instance [2,47, 62,64]. These numerical 
schemes are based on a tailored mass-preserving distance functional 


DEPI[ By) = al (Zi(x + vB(x))(1 + 8, B) —Zo(x — vB(x))(1 — 9, B)) dx. 
2 


A notable difference to image registration problems is that there is no template and 
reference image relationship. The displacement is applied to images, but in opposite 
directions. Further, intensity modulations are applied to ensure mass-preservation. 


Regularization Because of the mass-preservation component invertibility of the 
geometrical transformations in (60) is required as has been already stressed in [15]. 
However, simplifications of the hyperelastic regularization functional S"Y?* arise, 
since displacements are restricted in one direction. Thus the field inhomogeneity B 
needs to satisfy 
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0< (1+ 0,B) and0 < (1-—0,B) @ -1 <0,B <1. (61) 


This motivates using the following regularizer 


EPI = @ 2 
So [B] = 5 fava dx +B | wO.B)dx. 


The first summand is a diffusion term that ensures smoothness as commonly used 
and suggested by the forward model; see [22]. The convex penalty function 


y+ 


v) = ————_ 
¥0) (v-1lWvt+]) 
has singularities at +1 and ensures 0,B € L. This is essential to ensure positivity of 
the intensity modulations. It is shown in [62] that this regularizer ensures existence 
of solutions and that solutions satisfy (61). 


Numerical Implementation Numerical discretization of the reversed gradient can 
be done using a nodal grid for the field inhomogeneity and a finite volume approach. 
Intensity modulations are thus approximated by monitoring volume changes. Note 
that no coupling between the components of the Jacobian matrix is present since 
displacements are restricted along tv. A Matlab implementation is available as 
an extension to the FAIR toolbox [55] and integrated into the toolbox Statistical 
Parametric Mapping (SPM) [63]. 


Correction Results 
Figure 10 illustrates the correction method for EPI data of the brain of a healthy 
volunteer; see [62] for more results. Data courtesy by Harald Kugel, Department for 
Radiology, University of Miinster, Germany. The distance between the images 7 
and Z, with respect to DF”! is reduced considerably by the estimated transformation. 
Figure 10 shows the improvement of spatial correspondence between the dis- 
torted EPI data and an anatomical T1l-weighted MRI which can be assumed to be 
free of distortions. An axial slice zoomed into the frontal brain region is shown. 
Superimposed contour lines representing a white matter segmentation obtained from 
the anatomical image data are depicted in all subplots. Geometrical mismatch of 
the uncorrected EPI measurements and the anatomical data is most pronounced in 
frontal regions and around the Corpus Callosum. After correction, the geometrical 
correspondence improves considerably. 


6 Conclusion 
Image registration is an essential task in a variety of areas involving imaging 


techniques. This chapter presented a comprehensive overview of mathematical 
techniques used for nonlinear image registration. Emphasis was on regularization 


2048 L. Ruthotto and J. Modersitzki 


techniques that ensure a mathematically sound formulation of the problem, allow 
stable and fast numerical solution, and favor solutions that are realistic for the 
application in mind. 

Starting out from one of the most commonly used linear elastic model [10, 55], 
its limitations and extensions to nonlinear regularization functionals based on the 
theory of hyperelastic materials were discussed. A detailed overview of the available 
theoretical results was given. Insight into the existence theory of hyperelastic 
image registration problems was given and a state-of-the-art numerical scheme 
is presented. Finally, the potential of hyperelastic image registration for real-life 
medical imaging applications was outlined in two case studies. 


Cross-References 
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Optical Flow 
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Abstract 


We begin with traditional source detection algorithms in astronomy. We then 
introduce the sparsity data model. The starlet wavelet transform serves as our 
main focus in this article. Sparse modeling and noise modeling are described. 
Applications to object detection and characterization, and to image filtering 
and deconvolution, are discussed. The multiscale vision model is a further 
development of this work, which can allow for image reconstruction when the 
point spread function is not known or not known well. Bayesian and other 
algorithms are described for image restoration. A range of examples is used to 
illustrate the algorithms. 


1 Introduction 


Data analysis is becoming more and more important in astronomy. This can be 
explained by detector evolution, which concerns all wavelengths. In the 1980s, 
CCD (charge-coupled device) images had a typical size of 512 x 512 pixels, while 
astronomers now have CCD mosaics with 16,000 x 16,000 pixels or even more. 
At the same time, methods of analysis have become much more complex, and the 
human and financial efforts to create and process the data can sometimes be of the 
same order as for the construction of the instrument itself. As an example, for the 
ISOCAM camera of the Infrared Space Observatory (ISO), the command software 
of the instrument, and the online and offline data processing, required altogether 
70 person years of development, while 200 person years were necessary for the 
construction of the camera. The data analysis effort for the PLANCK project is 
even larger. Furthermore, the quantity of outputs requires the use of databases, and in 
parallel sophisticated tools are needed to extract ancillary astrophysical information, 
generally now through the web. From the current knowledge, new questions emerge, 
and it is necessary to proceed to new observations of a given object or a part of 
the sky. The acquired data need to be calibrated prior to useful information for the 
scientific project being extracted. 
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Data analysis acts during the calibration, the scientific information extraction 
process, and the database manipulation. The calibration phase consists of correcting 
various instrumental effects, such as the dark current (i.e., in the absence of any light, 
the camera does not return zero values, and the measured image is called the dark 
image and needs to be subtracted from any observation) or the flat-field correction 
(i.e., for uniform light, the detector does not return the same value for each pixel, and 
a normalization needs to be performed by dividing the observed image by the “flat” 
image). Hence, it is very important to know well the parameters of the detector (flat- 
field image, dark image, etc.), because any error on these parameters will propagate 
to the measurements. Other effects can also be corrected during this phase, such as 
the removal of the cosmic ray impacts or the field distortion (the pixel surface for 
each pixel does not correspond to the same surface on the sky). Depending on the 
knowledge of the instrument, each of these tasks may be more or less difficult. 

Once the data are calibrated, the analysis phase can start. Following the scientific 
objectives, several kinds of information can be extracted from the data, such as 
the detection of stars and galaxies, the measurement of their position, intensity, 
and various morphological parameters. The results can be compared to existing 
catalogs, obtained from previous observations. It is obviously impossible to cite 
all operations we may want to carry through on an astronomical image, and we 
have just mentioned the most common. In order to extract the information, it is 
necessary to take into account noise and point spread function. Noise is the random 
fluctuation which is added to the CCD data and comes partially from the detector 
and partially from the data. In addition to the errors induced by the noise on 
the different measurements, noise also limits the detection of objects and can be 
responsible for false detections. The point spread function is manifested in how the 
image of a star, for example, is generally spread out on several pixels, caused by 
the atmosphere’s effect on the light path. The main effect is a loss of resolution, 
because two sufficiently close objects cannot be separated. Once information has 
been extracted, such details can be compared to our existing knowledge. This 
comparison allows us to validate or reject our understanding of the universe. 

In this chapter, we will discuss in detail how to detect objects in astronomical 
images and how to take into account the point spread function through the 
deconvolution processing. 


Source Detection 


As explained above, source (i.e., object) extraction from images is a fundamental 
step for astronomers. For example, to build catalogs, stars and galaxies must be 
identified and their position and photometry must be estimated with good accuracy. 
Catalogs comprise a key result of astronomical research. Various methods have 
been proposed to support the construction of catalogs. One of the now most widely 
used software packages is SExtractor [6], which is capable of handling very large 
images. A standard source detection approach, such as in SExtractor, consists of the 
following steps: 
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Fig. 1 Example of Flux 
astronomical data: a point 
source and an extended 
source are shown, with noise 
and background. The 
extended object, which can be 
detected by eye, is undetected 
by a standard detection 
approach 
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. Background estimation. 

. Convolution with a mask. 
. Detection. 

. Deblending/merging. 

. Photometry. 

. Classification. 
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These different steps are described in the next section. Astronomical images 
contain typically a large set of point-like sources (the stars), some quasi point- 
like objects (faint galaxies, double stars), and some complex and diffuse structures 
(galaxies, nebulae, planetary stars, clusters, etc.). These objects are often hierarchi- 
cally organized: a star in a small nebula, itself embedded in a galaxy arm, itself 
included in a galaxy, and so on. 

The standard approach, which is presented in detail in Sect.2, presents some 
limits, when we are looking for faint extended objects embedded in noise. Figure 1 
shows a typical example where a faint extended object is under the detection limit. 
In order to detect such objects, more complex data modeling needs to be defined. 
Section 3 presents another approach to model and represent astronomical data, 
by using a sparse model in a wavelet dictionary. A specific wavelet transform, 
called the starlet transform or the isotropic undecimated wavelet transform, is 
presented. Based on this new modeling, several approaches are proposed in Sects. 4 
and 5. 


2 Standard Approaches to Source Detection 


We describe here the most popular way to create a catalog of galaxies from 
astronomical images. 
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The Traditional Data Model 


The observed data Y can be decomposed into two parts, the signal X and the 
noise NV: 


Y[k.l] = X[k.) + NIk./] (1) 


The imaging system can also be considered. If it is linear, the relation between the 
data and the image in the same coordinate frame is a convolution: 


Y [k,l] = (AX)[K. 1] + NIK, 1] (2) 


where H is the matrix related to the point spread function (PSF) of the imaging 
system. 

In most cases, objects of interest are superimposed on a relatively flat signal B, 
called background signal. The model becomes 


Y [k, 1] = (AX)[k, 1] + Blk. 1] + NIK. TU] (3) 


PSF Estimation 


The PSF H can be estimated from the data or from an optical model of the 
imaging telescope. In astronomical images, the data may contain stars, or one can 
point towards a reference star in order to reconstruct a PSF. The drawback is the 
“degradation” of this PSF because of unavoidable noise or spurious instrument 
signatures in the data. So, when reconstructing a PSF from experimental data, one 
has to reduce very carefully the images used (background removal for instance). 
Another problem arises when the PSF is highly variable with time, as is the case 
for adaptive optics (AO) images. This means usually that the PSF estimated when 
observing a reference star, after or before the observation of the scientific target, has 
small differences from the perfectly correct PSF. 

Another approach consists of constructing a synthetic PSF. Various studies [11, 
21,38, 39] have suggested a radially symmetric approximation to the PSF: 


r2 —B 
P(r)« (: + 7) (4) 


The parameters f and R are obtained by fitting the model with stars contained in 
the data. 

In the case of AO systems, this model can be used for the tail of the PSF (the 
so-called seeing contribution), while in the central region, the system provides an 
approximation of the diffraction-limited PSF. The quality of the approximation is 
measured by the Strehl ratio (SR), which is defined as the ratio of the observed peak 
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intensity in the image of a point source to the theoretical peak intensity of a perfect 
imaging system. 


Background Estimation 


The background must be accurately estimated; otherwise it will introduce bias in 
flux estimation. In [7,28], the image is partitioned into blocks, and the local sky level 
in each block is estimated from its histogram. The pixel intensity histogram p(Y) is 
modeled using three parameters, the true sky level B, the RMS (root mean square) 
noise o, and a parameter describing the asymmetry in p(Y) due to the presence of 
objects, and is defined by [7]: 


pY) = “exp (0? /2a”) exp [—(Y — B)/a]erfc (2 — “*) (5) 

Median filtering can be applied to the 2D array of background measurements 

in order to correct for spurious background values. Finally the background map 

is obtained by a bilinear or a cubic interpolation of the 2D array. The block size 

is a crucial parameter. If it is too small, the background estimation map will be 

affected by the presence of objects, and if too large it will not take into account real 
background variations. 

In [6, 15], the local sky level is calculated differently. A 3-sigma clipping around 
the median is performed in each block. If the standard deviation is changed by less 
than 20% in the clipping iterations, the block is uncrowded, and the background 
level is considered to be equal to the mean of the clipped histogram. Otherwise, 
it is calculated by c; x median — cp X mean, where c} = 3,c2 = 2 in [15] and 
c, = 2.5,c2 = 1.5 in [6]. This approach has been preferred to histogram fitting for 
two reasons: it is more efficient from the computation point of view and more robust 
with small sample size. 


Convolution 


In order to optimize the detection, the image must be convolved with a filter. The 
shape of this filter optimizes the detection of objects with the same shape. Therefore, 
for star detection, the optimal filter is the PSF. For extended objects, a larger filter 
size is recommended. In order to have optimal detection for any object size, the 
detection must be repeated several times with different filter sizes, leading to a kind 
of multiscale approach. 


Detection 


Once the image is convolved, all pixels Y [k, /] at location (k,/) with a value larger 
than T[k,/] are considered as significant, i.e., belonging to an object. T[k,/] is 
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generally chosen as B[k,/] + Ko, where B[k, /] is the background estimate at the 
same position, o is the noise standard deviation, and K is a given constant (typically 
chosen between 3 and 5). The thresholded image is then segmented, i.e., a label is 
assigned to each group of connected pixels. The next step is to separate the blended 
objects which are connected and have the same label. 

An alternative to the thresholding/segmentation procedure is to find peaks. This 
is only well suited to star detection and not to extended objects. In this case, the next 
step is to merge the pixels belonging to the same object. 


Deblending/Merging 


This is the most delicate step. Extended objects must be considered as single 
objects, while multiple objects must be well separated. In SExtractor, each group of 
connected pixels is analyzed at different intensity levels, starting from the highest 
down to the lowest level. The pixel group can be seen as a surface, with mountains 
and valleys. At the beginning (highest level), only the highest peak is visible. When 
the level decreases, several other peaks may become visible, defining therefore 
several structures. At a given level, two structures may become connected, and 
the decision whether they form only one (i.e., merging) or several objects (i.e., 
deblending) must be taken. This is done by comparing the integrated intensities 
inside the peaks. If the ratio between them is too low, then the two structures must 
be merged. 


Photometry and Classification 


Photometry 

Several methods can be used to derive the photometry of a detected object 
[7,29]. Adaptive aperture photometry uses the first image moment to determine 
the elliptical aperture from which the object flux is integrated. Kron [29] proposed 
an aperture size of twice the radius of the first image moment radius r;, which leads 
to recovery of most of the flux (>90%). In [6], the value of 2.5r; is discussed, 
leading to loss of less than 6 % of the total flux. Assuming that the intensity profiles 
of the faint objects are Gaussian, flux estimates can be refined [6,35]. When the 
image contains only stars, specific methods can be developed which take the PSF 
into account [18, 42]. 


Star-Galaxy Separation 
In the case of star—-galaxy classification, following the scanning of digitized images, 
Kurtz [30] lists the following parameters which have been used: 


1. Mean surface brightness; 
2. Maximum intensity and area; 
3. Maximum intensity and intensity gradient; 
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4. Normalized density gradient; 
5. Areal profile; 
6. Radial profile; 
7. Maximum intensity, 2nd and 4th order moments, and ellipticity; 
8. The fit of galaxy and star models; 
9. Contrast versus smoothness ratio; 
10. The fit of a Gaussian model; 
11. Moment invariants; 
12. Standard deviation of brightness; 
13. 2nd order moment; 
14. Inverse effective squared radius; 
15. Maximum intensity and intensity-weighted radius; 
16. 2nd and 3rd order moments, number of local maxima, and maximum intensity. 


References for all of these may be found in the cited work. Clearly there is 
room for differing views on parameters to be chosen for what is essentially the 
same problem. It is of course the case also that aspects such as the following will 
help to orientate us towards a particular set of parameters in a particular case: the 
quality of the data; the computational ease of measuring certain parameters; the 
relevance and importance of the parameters measured relative to the data analysis 
output (e.g., the classification, or the planar graphics); and, similarly, the importance 
of the parameters relative to theoretical models under investigation. 


Galaxy Morphology Classification 

The inherent difficulty of characterizing spiral galaxies especially when not face-on 
has meant that most work focuses on ellipticity in the galaxies under study. This 
points to an inherent bias in the potential multivariate statistical procedures. In the 
following, it will not be attempted to address problems of galaxy photometry per se 
[17,44], but rather to draw some conclusions on what types of parameters or features 
have been used in practice. 

From the point of view of multivariate statistical algorithms, a reasonably 
homogeneous set of parameters is required. Given this fact, and the available 
literature on quantitative galaxy morphological classification, two approaches to 
parameter selection appear to be strongly represented: 


1. The luminosity profile along the major axis of the object is determined at 
discrete intervals. This may be done by the fitting of elliptical contours, followed 
by the integrating of light in elliptical annuli [33]. A similar approach was 
used in the ESO-Uppsala survey. Noisiness and faintness require attention to 
robustness in measurement: the radial profile may be determined taking into 
account the assumption of a face-on optically thin axisymmetric galaxy and may 
be further adjusted to yield values for circles of given radius [64]. Alternatively, 
isophotal contours may determine the discrete radial values for which the profile 
is determined [62]. 
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2. Specific morphology-related parameters may be derived instead of the profile. 
The integrated magnitude within the limiting surface brightness of 25 or 26 mag. 
arcsec” in the visual is popular [33, 61]. The logarithmic diameter (D2,) is 
also supported by Okamura [43]. It may be interesting to fit to galaxies under 
consideration model bulges and disks using, respectively, rior exponential laws 
[62], in order to define further parameters. Some catering for the asymmetry of 
spirals may be carried out by decomposing the object into octants; furthermore, 
the taking of a Fourier transform of the intensity may indicate aspects of the 
spiral structure [61]. 


The following remarks can be made relating to image data and reduced data: 


e The range of parameters to be used should be linked to the subsequent use to 
which they might be put, such as to underlying physical aspects. 

¢ Parameters can be derived from a carefully constructed luminosity profile, rather 
than it being possible to derive a profile from any given set of parameters. 

¢ The presence of both partially reduced data such as luminosity profiles, and more 
fully reduced features such as integrated flux in a range of octants, is of course 
not a hindrance to analysis. However, it is more useful if the analysis is carried 
out on both types of data separately. 


Parameter data can be analyzed by clustering algorithms, by principal component 
analysis, or by methods for discriminant analysis. Profile data can be sampled at 
suitable intervals and thus analyzed also by the foregoing procedures. It may be 
more convenient in practice to create dissimilarities between profiles and analyze 
these dissimilarities: this can be done using clustering algorithms with dissimilarity 
input. 


3 Mathematical Modeling 


Different models may be considered to represent the data. One of the most effective 
is certainly the sparsity model, especially when a specific wavelet dictionary is 
chosen to represent the data. We introduce here the sparsity concept as well as the 
wavelet transform decomposition, which is the most used in astronomy. 


Sparsity Data Model 


A signal X, X = [x1,..., xv]', is sparse if most of its entries are equal to zero. For 
instance, a k-sparse signal is a signal where only k samples have a nonzero value. A 
less strict definition is to consider a signal as weakly sparse or compressible when 
only a few of its entries have a large magnitude, while most of them are close to 
zero. 
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If a signal is not sparse, it may be sparsified using a given data representation. 
For instance, if X is a sine, it is clearly not sparse but its Fourier transform is 
extremely sparse (i.e., 1-sparse). Hence, we say that a signal X is sparse in the 
Fourier domain if its Fourier coefficients X [uv], x [u] = v pp aes X [k]e! a 
are sparse. More generally, we can model a vector signal X ¢€ RY” as the 
linear combination of T elementary waveforms, also called signal atoms: X = 
Pa = Sg a[i]¢;, where a[i] = (X,¢;) are called the decomposition coef- 
ficients of X in the dictionary ® = [¢,...,¢7] (the N x T matrix whose 
columns are the atoms normalized to a unit €2-norm, ie., Vi € [1,T], ||@illo = 
1). 

Therefore, to get a sparse representation of our data, we need first to define the 
dictionary ® and then to compute the coefficients @. x is sparse in ® if the sorted 
coefficients in decreasing magnitude have fast decay, i.e., most coefficients @ vanish 
except for a few. 

The best dictionary is the one which leads to the sparsest representation. Hence, 
we could imagine having a huge overcomplete dictionary (i.e, T >> MN), but 
we would be faced with prohibitive computation time cost for calculating the 
a coefficients. Therefore, there is a trade-off between the complexity of our 
analysis step (i.e., the size of the dictionary) and the computation time. Some 
specific dictionaries have the advantage of having fast operators and are very good 
candidates for analyzing the data. 

The Isotropic Undecimated Wavelet Transform (IUWT), also called starlet 
wavelet transform, is well known in the astronomical domain because it is well 
adapted to astronomical data where objects are more or less isotropic in most cases 
[54,57]. For most astronomical images, the starlet dictionary is very well adapted. 


The Starlet Transform 


The starlet wavelet transform [53] decomposes ann x n image Co into a coefficient 
set W = {w),...,wy, cy}, as a superposition of the form 


J 
colk. 1] = cs[k. I + D> wilk. 1), 


j=l 


where c; is a coarse or smooth version of the original image cp and w; represents 
the details of cg at scale 2~/ (see Starck et al. [56,58] for more information). Thus, 
the algorithm outputs J + 1 sub-band arrays of size N x N. (The present indexing 
is such that 7 = 1 corresponds to the finest scale or high frequencies.) 7 

The decomposition is achieved using the filter bank (A2p, gap = 6 — hop, hop = 
6, Zap = 5), where hyp is the tensor product of two 1D filters hjp and 6 is the Dirac 
function. The passage from one resolution to the next one is obtained using the “a 
trous” (“with holes”) algorithm [58]: 
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Fig. 2 Left, the cubic spline function @; right, the wavelet y 


cjtilk. 1] = > Yo hl] hin] cj [k + 2/m,1 + 2/7), 
mon (6) 
w ilk, J] = c[k, 1] —cj4ilk, J] 5 


If we choose a B3-spline for the scaling function, 


(x) = B3(x) = 


1 
gl *¥-2P 4] x-1P +6] xP -4/x +1 P+] 2421) (7) 


the coefficients of the convolution mask in one dimension are jp = 
{2 131 


1 : . : 
Te-a°Ra? ae and in two dimensions, 


hop = (1/16 1/4 3/8 1/4 1/16) 
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Figure 2 shows the scaling function and the wavelet function when a cubic spline 
function is chosen as the scaling function @. 

The most general way to handle the boundaries is to consider that c[k + N] = 
c[N —k] (“mirror”). But other methods can be used such as periodicity (c[k +N] = 
c[N]) or continuity (c[k + N] = clk]). 

The starlet transform algorithm is: 


1. We initialize j to 0 and we start with the data c ;[k, /]. 
2. We carry out a discrete convolution of the data c;[k,/] using the filter (hap), 
using the separability in the two-dimensional case. In the case of the B3- 
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Fig. 3 Wavelet transform of NGC 2997 by the IUWT. The co-addition of these six images 
reproduces exactly the original image 


spline, this leads to a row-by-row convolution with (4: ie 3, i i): followed 
by column-by-column convolution. The distance between the central pixel and 
the adjacent ones is 2/. 

3. After this smoothing, we obtain the discrete wavelet transform from the differ- 
ence c;[k, 1] — cj+i]k, 1]. 

4. If 7 is less than the number J of resolutions we want to compute, we increment 
j and then go to step 2. 


5. The seta = {w,,...,w ,c,s} represents the wavelet transform of the data. 


This starlet transform is very well adapted to the detection of isotropic features, 
and this explains its success for astronomical image processing, where the data 
contain mostly isotropic or quasi-isotropic objects, such as stars, galaxies, or galaxy 
clusters. 

Figure 3 shows the starlet transform of the galaxy NGC 2997 displayed in Fig. 4. 
Five wavelet scales and the final smoothed plane (lower right) are shown. The 
original image is given exactly by the sum of these six images. 


The Starlet Reconstruction 
The reconstruction is straightforward. A simple co-addition of all wavelet scales 


reproduces the original map: co|k,/] = cy[k, 7] + = w,|k,/]. But because the 
transform is non-subsampled, there are many ways to reconstruct the original image 
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Fig. 4 Galaxy NGC 2997 


from its wavelet transform [53]. For a given wavelet filter bank (h, g), associated 
with a scaling function ¢@ and a wavelet function y, any synthesis filter bank (1, g), 
which satisfies the following reconstruction condition 


i* (vyh(v) + &* (V8) = 1, (8) 


leads to exact reconstruction. For instance, for isotropic h, if we choose h=h (the 
synthesis scaling function ¢ = @), we obtain a filter ¢ defined by [53] 


g=st+h. 
If / is a positive filter, then g is also positive. For instance, if jp = [1,4, 6,4, 1]/16, 
then ip = [1, 4, 22, 4, 1]/16. That is, 1p is positive. This means that g is no longer 


related to a wavelet function. The 1D detail synthesis function related to gip is 
defined by 


1 - t 1 t 

= -~j= t = —]}. 9 

5 Vip (5) dip(t) + hal (5) (9) 
Note that by choosing dio = ¢)p, any synthesis function Wip which satisfies 


ip vy 2v) = $3(v) — 8220) (10) 
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Fig. 5 Left, dy p the 1D synthesis scaling function and right, Wh p the 1D detail synthesis function 


leads to an exact reconstruction [36] and Wip (0) can take any value. The synthesis 
function Wip does not need to verify the admissibility condition (i.e., to have a zero 
mean). 

Figure 5 shows the two functions éiv(= ¢ip) and Wip used in the reconstruction 
in 1D, corresponding to the synthesis filters hp = hyp and Zip = 6 + hyp. More 
details can be found in [53]. 


Starlet Transform: Second Generation 


Fin)—Fip2v) 
giv(v) 
leads to a filter gp equal to 6 — hyp * hyp. In this case, the synthesis function Yip 

is defined by iWin) = ¢ p(t), and the filter gp = 4 is the solution to (8). 

We end up with a synthesis scheme where only the smooth part is convolved 
during the reconstruction. 

Deriving fh from a spline scaling function, for instance B,(h, = [1,2, 1]/4) or 
B; (h3 = [1,4, 6,4, 1]/16) (note that h3 = h, « h1), since hip is even-symmetric 
(i.e., H(z) = H(z_')), the z-transform of gip is then 


iggy? 
1-w@=1-7(2) 


I 
556 (—2! — 82) — 282" — 562 + 186 — 562 ' 287-7 — 823-74), 
(11) 


A particular case is obtained when dip = éip and Wip(2v) — , which 


G(z) 


II 


which is the z-transform of the filter 


Zip = [—1, —8, —28, —56, 186, —56, —28, —8, —1]/256. 
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We get the following filter bank: 


hip =h3 =h = [1,4,6,4, 1]/16 
gip = d—h«h = [-1, 8, —28, —56, 186, —56, —28, -8,—1]/256 (12) 
Zip => 6 . 


The second-generation starlet transform algorithm is: 


. We initialize j to 0 and we start with the data c ;[k]. 
. We carry out a discrete convolution of the data c;[k] using the filter 4jp. The 
distance between the central pixel and the adjacent ones is 2/ . We obtain c ; +1 [k]. 

. We do exactly the same convolution on c ;+,[k] and we obtain ci 41 [k]. 

4. After this two-step smoothing, we obtain the discrete starlet wavelet transform 
from the difference wj41[k] = c;[k] — cj +, [k]. 

5. If j is less than the number J of resolutions we want to compute, we increment 
j and then go to step 2. 

6. The seta = {w),...,w , cs} represents the starlet wavelet transform of the data. 


NOR 


WwW 


As in the standard starlet transform, extension to 2D is trivial. We just replace 
the convolution with /;p by a convolution with the filter 42p, which is performed 
efficiently by using the separability. 

With this specific filter bank, there is a no convolution with the filter g}p during 
the reconstruction. Only the low-pass synthesis filter hyp is used. 

The reconstruction formula is 


ef] = AY «cj 4) + willl, (13) 


and denoting L/ = h© *---*hY—) and L° = 6, we have 


J 


coll] = (L’ * es) [+ 55 (LI #w,) (14) 


j=l 


Each wavelet scale is convolved with a low-pass filter. 
The second-generation starlet reconstruction algorithm is: 


1. The seta = {w,...,wys, cs} represents the input starlet wavelet transform of 
the data. 

2. We initialize j to J — 1 and we start with the coefficients c ;[k]. 

3. We carry out a discrete convolution of the data c ; +; [k] using the filter (1p). The 
distance between the central pixel and the adjacent ones is 2/. We obtain ci 4, lk]. 

4. Compute cj [k] = ci, ,[k] + wj+ilk]. 
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Fig. 6 Left, the ¢; dD. analysis scaling function and right, the Wp analysis wavelet function. The 
synthesis functions ¢;p and yp are the same as those in Fig. 5 


5. If 7 is larger than 0, 7 = j — | and then go to step 3. 
6. co contains the reconstructed data. 


As for the transformation, the 2D extension consists just in replacing the convolution 
by /ip with a convolution by hap. 

Figure 6 shows the analysis scaling and wavelet functions. The synthesis 
functions dw and Wip are the same as those in Fig.5. As both are positive, we 
have a decomposition of an image X on positive scaling functions p)p and Wp, but 
the coefficients @ are obtained with the starlet wavelet transform and have a zero 
mean (except for c;), as a regular wavelet transform. 

In 2D, similarly, the second-generation starlet transform leads to the representa- 
tion of an image X[k, /]: 


J 
X[k = Soe (mnyesfn.n} + DOYS Oe m.nywjlm.n], (15) 


mn j=lman 


where $j p,(m.n) = 2-7 dp(24(k — m))b(2/ (I — n)) and Fh (m,n) = 
2-4 Wip(2/ (k — m))Wip(2 (I — n)). 

¢ and 6 are positive, and w j are zero mean 2D wavelet coefficients. 

The advantage of the second-generation starlet transform will be seen in sec- 
tion “Sparse Positive Decomposition” below. 


Starlet Transform in Astronomical Data Processing 2069 


Sparse Modeling of Astronomical Images 


Using the sparse modeling, we now consider that the observed signal X can be 
considered as a linear combination of a few atoms of the wavelet dictionary ® = 
[o1,..., br]. The model of Eq. 3 is then replaced by the following: 


Y=H®a+N+B8B (16) 


and X = a, anda = {w,,...,wy,c7}. Furthermore, most of the coefficients 
a will be equal to zero. Positions and scales of active coefficients are unknown, 
but they can be estimated directly from the data Y. We define the multiresolution 
support M of an image Y by 


1 ifw,[k, /] is significant 


[k,l] = 
MMs | 0 if w;[k,/] is not significant 


(17) 


where w ; [k, /] is the wavelet coefficient of Y at scale j and at position (k,/). Hence, 
M describes the set of active atoms in Y. If H is compact and not too extended, then 
M describes also well the active set of X. This is true because the background B is 
generally very smooth, and therefore, a wavelet coefficient w ; [k, /] of Y , which does 
not belong to the coarsest scale, is only dependent on X and N (the term < ¢;, B > 
being equal to zero). 


Selection of Significant Coefficients Through Noise Modeling 

We need now to determine when a wavelet coefficient is significant. Wavelet 
coefficients of Y are corrupted by noise, which follows in many cases a Gaussian 
distribution, a Poisson distribution, or a combination of both. It is important to detect 
the wavelet coefficients which are “significant,” i.e., the wavelet coefficients which 
have an absolute value too large to be due to noise. 

For Gaussian noise, it is easy to derive an estimation of the noise standard 
deviation o; at scale j from the noise standard deviation, which can be evaluated 
with good accuracy in an automated way [55]. To detect the significant wavelet 
coefficients, it suffices to compare the wavelet coefficients w;[k,/] to a threshold 
level t;. t; is generally taken equal to Ko;, and K, as noted in Sect. 2, is chosen 
between 3 and 5. The value of 3 corresponds to a probability of false detection of 
0.27 %. If w;[k,/] is small, then it is not significant and could be due to noise. If 
w[k, [] is large, it is significant: 


if |wj[k,/]|> ¢; then w;[k,/] is significant 


18 
if | w,[k,/] |<; then w,[k, /] is not significant (18) 


When the noise is not Gaussian, other strategies may be used: 
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Poisson noise: if the noise in the data Y is Poisson, the transformation [1] 
A(Y) = 2,/Y + - acts as if the data arose from a Gaussian white noise model, 
with o = 1, under the assumption that the mean value of Y is sufficiently large. 
However, this transform has some limits, and it has been shown that it cannot be 
applied for data with less than 20 counts (due to photons) per pixel. So for X-ray 
or gamma ray data, other solutions have to be chosen, which manage the case of 
a reduced number of events or photons under assumptions of Poisson statistics. 
Gaussian + Poisson noise: the generalization of variance stabilization [40] is 


G(Y[k,/]) = 2 Jorn. ie 0 +o2—ag 


where q@ is the gain of the detector and g and o are the mean and the standard 
deviation of the readout noise. 

Poisson noise with few events using the MS-VST: for images with very few 
photons, one solution consists in using the Multi-Scale Variance Stabilization 
Transform (MS-VST) [66]. The MS-VST combines both the Anscombe trans- 
form and the starlet transform in order to produce stabilized wavelet coefficients, 
i.e., coefficients corrupted by a Gaussian noise with a standard deviation equal to 
1. In this framework, wavelet coefficients are now calculated by 


Starlet 
4: Cj = Vin Ln Aivln|hivln] 
MS-VST cj—ifk + 2/7!m, 1 + 2/-'n] (19) 


w; = Aj-i1(cj-1) — Aj (c;) 


where A; is the VST operator at scale j defined by 


Aj (cj) = bY /|cj +e| (20) 


where the variance stabilization constants bY) and e) only depend on the 
filter 4}p and the scale level 7. They can all be precomputed once for any 
given /jp [66]. The multiresolution support is computed from the MS-VST 
coefficients, considering a Gaussian noise with a standard deviation equal to 1. 
This stabilization procedure is also invertible as we have 


J 
co = Ag! | As(as) + Yow; (21) 


j=l 


For other kinds of noise (correlated noise, nonstationary noise, etc.), other solutions 
have been proposed to derive the multiresolution support [57]. 
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Sparse Positive Decomposition 


Many astronomical images can be modeled as a sum of positive features, like stars 
and galaxies, which are more or less isotropic. The previous representation, based 
on the starlet transform, is well adapted to the representation of isotropic objects, 
but does not introduce any prior relative to the positivity of the features contained 
in our image. A positive and sparse modeling of astronomical images is similar to 
Eq. 16: 


Y=H@a+N+B (22) 
or 
Y=%a+N+B (23) 


if we do not take into account the point spread function. All coefficients in a 
are now positive, and all atoms in the dictionary ® are positive functions. Such 
a decomposition normally requires computationally intensive algorithms such as 
matching pursuit [37]. The second-generation starlet transform offers us a new 
way to perform such a decomposition. Indeed, we have seen in section “Starlet 
Transform: Second Generation” that, using a specific filter bank, we can decompose 
an image Y on a positive dictionary ® (see Fig. 5) and obtain a set of coefficients 
a), where vw”) = WY = {w),...,wy,cy}, W being the starlet wavelet 
transform operator. a@ coefficients are positive and negative and are obtained 
using the standard starlet wavelet transform algorithm. Hence, by thresholding all 
negative (respectively, positive) coefficients, the reconstruction is always positive 
(respectively, negative), since ® contains only positive atoms. 

Hence, we would like to have a sparse set of positive coefficients @ which verify 
®q = Y. But in order to take into account the background and the noise, we need 
to define the constraint in the wavelet space (i.e., W®a = WY = a)), and this 
constraint must be applied only to the subset of coefficients in w) which are larger 
than the detection level. Therefore, to get a sparse positive decomposition on ®, we 
need to minimize 


&=min|la |, st. MWea = Mo” , (24) 


where M is the multiresolution support defined in the previous section (i.e., 
M,[k, 1] = 1 if a significant coefficient is detected at scale j and at position (k, /), 
and zero otherwise). To remove the background, we have to set M7+1[k,/] = 0 for 
all (k,/). 

It was shown that such optimization problems can be efficiently solved through 
an iterative soft thresholding (IST) algorithm [14, 24, 52]. The following algorithm, 
based on the IST, allows to take into account the noise modeling through the 
multiresolution support and force the coefficients to be all positive: 
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1. Taking the second-generation starlet wavelet transform of the data Y, we obtain 
(Y) 
av), 
2. From a given noise model, determine the multiresolution support M. 
3. Set the number of iterations Niter, the first threshold, A© = MAX(a?), and the 
solution @& = 0. 
4, For 0 = 1, Niter do: 


* Reconstruct the image Y” froma : YO = a, 

e Taking the second-generation starlet wavelet transform of the data Y®, we 
obtaina?” = Wea", 

* Compute the significant residual r“: 


Oa 47 (a 7 a”) = M (a — Woe) (25) 


* Calculate the value A“ = 2 (1 —i/Niter) 

¢ Update the solution, by adding the residual, applying a soft thresholding on 
positive coefficients using the threshold level A“, and setting all negative 
coefficients to zero. 


GHD = GO 4 7O — 40), eS 
= (@ + M (a — Wea) — 1) 
+ 


© i =i+. 
5. The set & = @ it") represents the sparse positive decomposition of the data. 


The threshold parameter A“ decreases with the iteration number, and it plays a role 
similar to the cooling parameter of the simulated annealing techniques, i.e., it allows 
the solution to escape from local minima. 


Example 1: Sparse Positive Decomposition of NGC 2997 
Figure 7 shows the positive starlet decomposition, using 100 iterations, and can be 
compared to Fig. 3. 


Example 2: Sparse Positive Starlet Decomposition of a Simulated Image 
The next example compares the standard starlet transform to the positive starlet 
decomposition (PSD) on a simulated image. 

Figure 8 shows respectively from top to bottom and left to right (a) the original 
simulated image, (b) the noisy data, (c) the reconstruction from the PSD coefficients, 
and (d) the residual between the noisy data and the PSD reconstructed image 
(i.e., image b-image c). Hence, the PSD reconstructed image gives a very good 
approximation of the original image. No structures can be seen in the residual, and 
all sources are well detected. 
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Fig. 7 Positive starlet decomposition of the galaxy NGC 2997 with six scales 


The first PSD scale does not contain any nonzero coefficient. Figure 9, top, shows 
the first four scales of the wavelet transform, and Fig. 9, bottom, the first four scales 
of the PSD. 


4 Source Detection Using a Sparsity Model 


As described is the previous section, the wavelet coefficients of Y which do not 
belong to the coarsest scale cy are not dependent on the background. This is 
a serious disadvantage, since the background estimation can be sometimes very 
problematic. 

Two approaches have been proposed to detect sources, assuming the signal is 
sparse in the wavelet domain. The first consists in first removing the noise and the 
background and then applying the standard approach described in Sect. 2. It has been 
used for many years for X-ray source detection [45,59]. The second approach, called 
Multiscale Vision Model [8], attempts to define directly an astronomical object in 
the wavelet space. 


Detection Through Wavelet Denoising 


The most commonly used filtering method is hard thresholding, which consists of 
setting to 0 all wavelet coefficients of Y which have an absolute value lower than a 
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Fig. 8 (a and b) Original simulated image and the same image contaminated by a Gaussian noise. 
(ce and d) Reconstructed image for the positive starlet coefficients of the noisy image using 50 
iterations, and residual (i.e., noisy image — reconstructed image) 


threshold f;: 


wjlk.1] if | wile.) [> tj 


w[k, 1] = 
ee 0 otherwise 


(27) 


More generally, for a given sparse representation (i.e., wavelet) with its associated 
fast transform W and fast reconstruction R, we can derive a hard threshold denoising 
solution X from the data Y, by first estimating the multiresolution support MW using 
a given noise model, and then calculating 


X =RMWY. (28) 


We transform the data, multiply the coefficients by the support, and reconstruct the 
solution. 

The solution can however be improved by considering the following optimization 
problem, miny || M(WY — WX) ||3, where M is the multiresolution support of Y. 


Fig. 9 Top, starlet transform, and bottom, positive starlet decomposition of a simulated astronom- 
ical image 
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A solution can be obtained using the Landweber iterative scheme [51,58]: 
Xx"t! = x" 4 RM [WY — WX"] (29) 


If the solution is known to be positive, the positivity constraint can be introduced 
using the following equation: 


X"+! = Py (X" + RM [WY — WX"]) (30) 


where P_ is the projection on the cone of nonnegative images. 

This algorithm allows us to constrain the residual to have a zero value within the 
multiresolution support [58]. For astronomical image filtering, iterating improves 
significantly the results, especially for the photometry (i.e., the integrated number 
of photons in a given object). 

Removing the background in the solution is straightforward. The algorithm does 
not need to be modified. We only need to set to zero the coefficients related to the 
coarsest scale in the multiresolution support: Vk M,[k,/] = 0. 


The Multiscale Vision Model 


Introduction 

The wavelet transform of an image Y by the starlet transform produces at each scale 
j aset {w;}. This has the same number of pixels as the image. The original image 
I can be expressed as the sum of all the wavelet scales and the smoothed array cz 
by the expression 
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J 
Y[k1] =cs{k. + Sw lk, 0). (1) 


j=l 


Hence, we have a multiscale pixel representation, i.e., each pixel of the input image 
is associated with a set of pixels of the multiscale transform. A further step is to 
consider a multiscale object representation, which would associate with an object 
contained in the data a volume in the multiscale transform. Such a representation 
obviously depends on the kind of image we need to analyze, and we present here a 
model which has been developed for astronomical data. It may however be used for 
other kinds of data, to the extent that such data are similar to astronomical data. We 
assume that an image Y can be decomposed into a set of components: 


No 
Yk = 0 X[k.) + Bik + NIK /] (32) 


i=1 


where N, is the number of components, X; are the components contained in the 
data (stars, galaxies, etc.), B is the background image, and WN is the noise. 

To perform such a decomposition, we have to detect, to extract, to measure, 
and to recognize the significant structures. This is done by first computing the 
multiresolution support of the image (i.e., the set of significant active coeffi- 
cients) and by applying a segmentation scale by scale. The wavelet space of 
a 2D direct space is a 3D volume. An object, associated with a component, 
has to be defined in this space. A general idea for object definition lies in the 
connectivity property. An object occupies a physical region, and in this region 
we can join any pixel to other pixels based on significant adjacency. Connectivity 
in direct space has to be transported into wavelet transform space. In order to 
define the objects, we have to identify the wavelet transform space pixels we 
can attribute to the objects. We describe in this section the different steps of this 
method. 


Multiscale Vision Model Definition 
The multiscale vision model, MVM [8], described an object as a hierarchical set of 
structures. It uses the following definitions: 


¢ Significant wavelet coefficient: a wavelet coefficient is significant when its 
absolute value is above a given detection limit. The detection limit depends on 
the noise model (Gaussian noise, Poisson noise, and so on). See section “Sparse 
Modeling of Astronomical Images” for more details. 

* Structure: a structure S; is a set of significant connected wavelet coefficients at 
the same scale /: 


Sik = {Wj [ky, 1], w; [k, b], 8 »Wj [pl pl} (33) 
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where p is the number of significant coefficients included in the structure Sj, 
and w;[x;, yi] is a wavelet coefficient at scale 7 and at position (x;, y;). 
¢ Object: an object is a set of structures: 


Or = {Sj ks e+ +s Sin kent (34) 


We define also the operator £ which indicates to which object a given structure 
belongs: £(S;x) = lis Sjx~ € Oj, and L(S;~) = 0 otherwise. 

¢ Object scale: the scale of an object is given by the scale of the maximum of its 
wavelet coefficients. 

¢ Interscale relation: the criterion allowing us to connect two structures into a 
single object is called the “interscale relation.” 

¢ Sub-object: a sub-object is a part of an object. It appears when an object has 
a local wavelet maximum. Hence, an object can be composed of several sub- 
objects. Each sub-object can also be analyzed. 


From Wavelet Coefficients to Object Identification 

Multiresolution Support Segmentation 

Once the multiresolution support has been calculated, we have at each scale a 
Boolean image (i.e., pixel intensity equals | when a significant coefficient has 
been detected, and 0 otherwise). The segmentation consists of labeling the Boolean 
scales. Each group of connected pixels having a “1” value gets a label value between 
1 and Linax, Lmax being the number of groups. This process is repeated at each scale 
of the multiresolution support. We define a “structure” S;,; as the group of connected 
significant pixels which has the label i at a given scale /. 


Interscale Connectivity Graph 

An object is described as a hierarchical set of structures. The rule which 
allows us to connect two structures into a single object is called “interscale 
relation.” Figure 10 shows how several structures at different scales are linked 
together and form objects. We have now to define the interscale relation. Let 
us consider two structures at two successive scales, S;; and S;+41 7. Each 
structure is located in one of the individual images of the decomposition 
and corresponds to a region in this image where the signal is significant. 
Denoting (Xm, Ym) the pixel position of the maximum wavelet coefficient value 
of S;x, Sj% is said to be connected to S;41, if Sj41) contains the pixel 
position (Xm, Vm) (i.e., the pixel position of the maximum wavelet coefficient 
of the structure $;, must also be contained in the structure $;+1)). Several 
structures appearing in successive wavelet coefficient images can be connected 
in such a way, which we call an object in the interscale connectivity graph. 
Hence, we identify n, objects in the wavelet space, each object O; being 
defined by a set of structures, and we can assign to each structure a label 
i, with i ¢€ [l,n.]: CUSjx) = 7 if the structure S;, belongs to the 
ith object. 
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Fig. 10 Example of connectivity in wavelet space: contiguous significant wavelet coefficients 
form a structure, and following an interscale relation, a set of structures forms an object. Two 
structures S;,.S;+41 at two successive scales belong to the same object if the position pixel of the 
maximum wavelet coefficient value of S; is included in $+ 


Filtering 

Statistically, some significant structures can be due to the noise. They contain very 
few pixels and are generally isolated, i.e., connected to no field at upper and lower 
scales. So, to avoid false detection, the isolated fields can be removed from the initial 
interscale connection graph. Structures at the border of the images may also have 
been detected because of the border problem and can be removed. 


Merging/Deblending 

As in the standard approach, true objects which are too close may generate a set of 
connected structures, initially associated with the same object, and a decision must 
be taken whether to consider such a case as one or two objects. Several cases may 
be distinguished: 


¢ Two (or more) close objects, approximately of the same size, generate a set of 
structures. At a given scale j, two separate structures S; and Sj are detected, 
while at the scale j + 1, only one structure is detected S$; +1,;, which is connected 
to the S;, and Sj». 

¢ Two (or more) close objects of different sizes generate a set of structures, from 
scale j to scale k (k > j). 
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In the wavelet space, the merging/deblending decision will be based on the local 
maxima values of the different structures belonging to this object. A new object 
(i.e., deblending) is derived from the structure S;, if there exists at least one other 
structure at the same scale belonging to the same object (i.e., there exists one 
structure S;+41,, and at least one structure S$; such that £(S;414) = L(Sj;5) = 
L£(S;,«)) and if the following relationship is verified: wi > wi_, and wi > wi'41, 
where: 


* wis the maximum wavelet coefficient of the structure Sj x: w'' = Max(Sj«): 

— wi_, = Oif Sj, is not connected to any structure at scale j — 1. 

- wey is the maximum wavelet coefficient of the structure S;—1,, where Sj—1, 
is such that £(S;-1) = L(S;,x) and the position of its highest wavelet 
coefficient is the closest to the position of the maximum of $;x. 

owes = Maxtwjtixpyis- +++ Wi+Lxnynt> Where all wavelet coefficients wj+1,x,y 


are at a position which belongs also to $j (i.€., Wj,x,y € Sj,k). 


When these conditions are verified, S$; and all structures at smaller scales which 
are directly or indirectly connected to S;, will define a new object. 


Object Identification 
We can now summarize this method allowing us to identify all the objects in a given 
image Y: 


1. We compute the wavelet transform with the starlet algorithm, which leads to a 
seta = WY = {w,...,wy,cz}. Each scale w; has the same size as the input 
image. 

. We determine the noise standard deviation in wy. 

. We deduce the thresholds at each scale from the noise modeling. 

. We threshold scale by scale and we do an image labeling. 

. We determine the interscale relations. 

. We identify all the wavelet coefficient maxima of the wavelet transform space. 

. We extract all the connected trees resulting from each wavelet transform space 
maximum. 


NYDN WNW 


Source Reconstruction 


Partial Reconstruction as an Inverse Problem 

A set of structures S; (Sj; = {Sj%,...,Sj74’}) defines an object O; which can be 
reconstructed separately from other objects, in order to provide the components X;. 
The co-addition of all reconstructed objects is a filtered version of the input data. We 
will denote a; the set of wavelet coefficients belonging to the object O;. Therefore, 
a; is a subset of the wavelet transform of X;, @; = WX;. Indeed, the last scale of 
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q@; is unknown, as well as many wavelet coefficients which have not been detected. 
Then the reconstruction problem consists of searching for an image X; such that 
its wavelet transform reproduces the coefficients a; (i.e., they are the same as those 
of S;, the detected structures). If W describes the wavelet transform operator and 
P,, the projection operator in the subspace of the detected coefficients (i.e., having 
set to zero all coefficients at scales and positions where nothing was detected), the 
solution is found by minimization of 


min || «; — Py (WX;) || (35) 


The size of the restored image X; is arbitrary, and it can be easily set greater than 
the number of known coefficients. It is certain that there exists at least one image X; 
which gives exactly q;, i.e., the original one. But generally we have an infinity of 
solutions, and we have to choose among them the one which is considered as correct. 
An image is always a positive function, which leads us to constrain the solution, but 
this is not sufficient to get a unique solution. More details on the reconstruction 
algorithm can be found in [8,57]. 


Examples 


Band Extraction 

We simulated a spectrum which contains an emission band at 3.50,.m and a 
nonstationary noise superimposed on a smooth continuum. The band is a Gaussian 
of width FWHM = 0.01 4m (FWHM = full width at half-maximum) and 
normalized such that its maximum value equals ten times the local noise standard 
deviation. 

Figure 11 (top) contains the simulated spectrum. The wavelet analysis results 
in the detection of an emission band at 3.50\1m above 30. Figure 11 (middle) 
shows the reconstruction of the detected band in the simulated spectrum. The real 
feature is overplotted as a dashed line. Figure 11 (bottom) contains the original 
simulation with the reconstructed band subtracted. It can be seen that there are no 
strong residuals near the location of the band, which indicates that the band is well 
reconstructed. The center position of the band, its FWHM, and its maximum can 
then be estimated via a Gaussian fit. More details about the use of MVM for spectral 
analysis can be found in [60]. 


Star Extraction in NGC 2997 

We applied MVM to the galaxy NGC 2997 (Fig. 12, top left). Two images were 
created by co-adding objects detected from scales 1 and 2 and from scales 3 to 6. 
They are displayed, respectively, in Fig. 12, top right and bottom left. Figure 12, 
bottom right, shows the difference between the input data and the image which 
contained the objects from scales | and 2. As we can see, all small objects have 
been removed, and the galaxy can be better analyzed. 
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Fig. 11 Top: simulated spectrum. Middle: reconstructed simulated band (full line) and original 
band (dashed line). Bottom: simulated spectrum minus the reconstructed band 


Galaxy Nucleus Extraction 
Figure 13 shows the extracted nucleus of NGC 2997 using the MVM method and 
the difference between the galaxy image and the nucleus image. 


5 Deconvolution 


Up to now, the PSF H has not been considered in the source detection. This means 
that all morphological parameters (size, ellipticity, etc.) derived from the detected 
objects need to be corrected from the PSF. Very close objects may also be seen 
as a single object because H acts as a blurring operator on the data. A solution 
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Fig. 12 (a) Galaxy NGC 2997, (b) objects detected from scales | and 2, (c) objects detected from 
scales 3 to 6, and (d) difference between (a) and (b) 


may consist in deconvolving first the data and carrying out the source detection 
afterwards. 

The problem of image deconvolution is ill-posed [3], and as a consequence, 
the matrix H modeling the imaging system is ill-conditioned. If Y is the 
observed image and X the unknown object, the equation HX = Y has 
not a unique and stable solution. Therefore, one must look for approximate 
solutions of this equation that are also physically meaningful. One approach 
is Tikhonov regularization theory [23]; however, a more general approach 
is provided by the so-called Bayes paradigm [25], even if it is applicable 
only to discrete problems. In this framework one can both take into account 
statistical properties of the data (Tikhonov regularization is obtained by assuming 
additive Gaussian noise) and also introduce a priori information on the unknown 
object. 
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Fig. 13 Upper left, galaxy NGC 2997; upper right, extracted nucleus; bottom, difference between 
the two previous images 


Statistical Approach to Deconvolution 


We assume that the detected image Y is the realization of a multivalued random 
variable J corresponding to the (unknown) value X of another multivalued random 
variable, the object O. Moreover, we assume that the conditional probability 
distribution p;(Y|X) is known. Since the unknown object appears as a set of 
unknown parameters, the problem of image deconvolution can be considered as a 
classical problem of parameter estimation. The standard approach is the maximum 
likelihood (ML) method. In our specific application, for a given detected image Y, 
this consists of introducing the likelihood function defined by 


Ly(X) = pi (Y;X) . (36) 
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Then the ML estimate of the unknown object is any maximizer X* of the likelihood 
function 


X*= Lyt{X G 
argmax y(X) , (37) 


if it exists. 

In our applications the likelihood function is the product of a very large number 
of terms (the data components are assumed to be statistically independent), so that 
it is convenient to take the logarithm of this function; moreover, if we consider its 
negative logarithm, the maximization problem is transformed into a minimization 
one. Let us consider the function 


Jo(X:¥) = —Aln Ly(X) +B , (38) 


where A, B are suitable constants. They are introduced in order to obtain a function 
which has a simple expression and is also nonnegative since, in our applications, 
the negative logarithm of the likelihood is bounded from below. Then, it is easy to 
verify that the problem of Eq. 37 is equivalent to the following one: 


x*= arg min Jo(X3Y) . (39) 


We consider now the model of Eq. 2 with three different examples of noise. 


Example 1. In the case of additive white Gaussian noise, by a suitable choice of the 
constants A, B, we obtain (we assume here that the background B is not subtracted 
even if it must be estimated) 


Jo(X;Y) = ||HX + B-Y||? , (40) 


and therefore, the ML approach coincides with the well-known least-squares (LS) 
approach. It is also well known that the function of Eq.40 is convex and strictly 
convex if and only if the equation HX = 0 has only the solution X¥ = 0. Moreover, 
it has always absolute minimizers, i.e., the LS problem has always a solution; but 
the problem is ill-conditioned because it is equivalent to the solution of the Euler 
equation: 


H'HX=H'(Y-B). (41) 


We remark that the ill-posedness of the LS problem is the starting point of Tikhonov 
regularization theory (see, for instance, [23,63]), and therefore, this theory is based 
on the tacit assumption that the noise affecting the data is additive and Gaussian. 
We remark that, in the case of object reconstruction, since objects are non- 
negative, we should consider the minimization of the function of Eq.40 on the 
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nonnegative orthant. With such a constraint the problem is not treatable in the 
standard framework of regularization theory. 


Example 2. In the case of Poisson noise, if we introduce the so-called generalized 
Kullback—Leibler (KL) divergence of a vector Z from a vector Y, defined by 


m Y; 
Dull.Z)= Yo}nin + Z:-¥| (42) 


i=l 
then, with a suitable choice of the constants A, B, the function Jo(X; Y) is given by 


Jo(X3Y) = Det(Y; HX + B) 


= So] ¥ in + (HX + By — yh 


i=1 


(43) 


It is quite natural to take the nonnegative orthant as the domain of this function. 
Moreover, it is well known that it is convex (strictly convex if the equation HX = 0 
has only the solution X = 0), nonnegative, and coercive. Therefore, it has absolute 
minimizers. However, these minimizers are strongly affected by noise, and the 
specific effect of the noise in this problem is known as checkerboard effect [41], 
since many components of the minimizers are zero. 


Example 3. In the case of Gauss+Poisson noise, the function Jo(X; Y) is given by 
a much more complex form. This function is also convex (strictly convex if the 
equation Hx = 0 has the unique solution x = 0), nonnegative, and coercive [2]. 
Therefore, it also has absolute minimizer on the nonnegative orthant. 


The previous examples demonstrate that, in the case of image reconstruction, ML 
problems are ill-posed or ill-conditioned. That means that one is not interested in 
computing the minimum points X* of the functions corresponding to the different 
noise models because they do not provide sensible estimates X of the unknown 
object. 

The previous remark is not surprising in the framework of inverse problem 
theory. Indeed it is generally accepted that, if the formulation of the problem does 
not use some additional information on the object, then the resulting problem is 
ill-posed. This is what happens in the maximum likelihood approach because we 
only use information about the noise with, possibly, the addition of the constraint of 
non-negativity. 

The additional information may consist, for instance, of prescribed bounds on the 
solution and/or its derivatives up to a certain order (in general not greater than two). 
These prescribed bounds can be introduced in the problem as additional constraints 
in the variational formulation provided by ML. However, in a quite natural 
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probabilistic approach, called the Bayesian approach, the additional information 
is given in the form of statistical properties of the object [25]. 

In other words, one assumes that the unknown object X is a realization of a 
vector-valued random variable O and that the probability distribution of O, the 
so-called prior denoted by po(X), is also known or can be deduced from known 
properties of the object. The most frequently used priors are Markov random fields 
or, equivalently, Gibbs random fields, i.e., they have the following form: 


1 
Po(X) = dia (44) 


where Z is a normalization constant, jz is a positive parameter (a hyperparameter 
in statistical language, a regularization parameter in the language of regularization 
theory), while 2(X) is a function, possibly convex. 

The previous assumptions imply that the joint probability density of the random 
variables O, I is given by 


Pow(X,Y) = pi(Y|X)polX) . (45) 


If we introduce the marginal probability density of the image J 


pit) = f porlX. Y) dX , (46) 


from Bayes’ formula we obtain the conditional probability density of O for a given 
value Y of I: 


Poul(X,Y) _ PI(Y |X) po(X) 


(47) 
p(X) P(X) 


Po(X|Y) = 


If in this equation we insert the detected value Y of the image, we obtain the a 
posteriori probability density of X: 


Po(X) 


Py(X) = X|Y) = Ly (X 
y(X) = po(X|Y) y ( mY) 


(48) 


Then, a maximum a posteriori (MAP) estimate of the unknown object is defined as 
any object X* that maximizes the a posteriori probability density: 


X*= argmax Py(X) . (49) 


As in the case of the likelihood, it is convenient to consider the negative 
logarithm of Py(X). If we assume a Gibbs prior as that given in Eq.44 and 
we take into account the definition of Eq.38, we can introduce the following 
function: 
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J(X;Y) =—AInPy(X)+ B-AlnZ 


(50) 
—Aln pi (VY) = Jo(XiY) + wIR(X) , 
where Jra(X) = AQ(X). Therefore, the MAP estimates are also given 
by 
xX = arg min J(X;Y) (51) 


and again one must look for the minimizers satisfying the non-negativity con- 
straint. 


The Richardson-Lucy Algorithm 


One of the most frequently used methods for image deconvolution in astronomy is 
an iterative algorithm known as the Richardson—Lucy (RL) algorithm [34, 48]. In 
emission tomography it is also denoted as expectation maximization (EM) because, 
as shown in [49], it can be obtained by applying to the ML problem with Poisson 
noise a general EM method introduced in [19] for obtaining ML estimates. 

In [49] it is shown that, if the iteration converges, then the limit is just an ML 
estimate in the case of Poisson data. Subsequently the convergence of the algorithm 
was proved by several authors in the case B = 0. An account can be found in [41]. 

The iteration is as follows: it is initialized with a positive image X (a constant 
array, in general); then, given X, X¥"*) is computed by 


Y 


xeaty = X™ AT ; 
HX +B 


(52) 


This algorithm has some nice features. First, the result of each iteration is auto- 
matically a positive array; second, in the case B = 0, the result of each iteration 
has the same flux of the detected image Y, and this property is interesting from the 
photometric point of view. 

The limit of the RL iteration is, in general, very noisy and sparse in 
pixel space (see the remark at the end of Example 2 in the previous section) 
and can provide satisfactory results in the case of star systems (see [4], 
section 3.1); in the case of complex systems, a reasonable solution can be 
obtained by a suitable stopping of the algorithm before convergence. This 
can be seen as a kind of regularization, and this property is called semi- 
convergence [3], i.e., the iteration first approaches the correct solution and then 
goes away. An example of RL reconstruction is shown in Fig. 14 (lower left 
panel). 

The main drawback of RL is that, in general, it is very slow and may require 
hundreds or thousands of iterations. The proposed acceleration approaches are based 
on the remark that RL is a scaled gradient method since it can be written in the 
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following form: 
XD = XM _ XOVy I (XY) , (53) 


where Vy denotes the gradient with respect to X and Jo(X; Y) is the data-fidelity 
function defined in Eq. 43. Therefore, a reduction of the number of iterations can be 
obtained by means of a suitable line search along the descent direction. This is the 
approach proposed by several authors. However, this structure of RL has inspired a 
recently proposed optimization method, known as scaled gradient projection (SGP) 
[10], which can be viewed as a general method for the constrained minimization of 
differentiable functions. 

If J(X) is the function to be minimized on the nonnegative orthant, then the 
method is based on the descent direction: 


D! = Py (X™ = 9,58" Vy I X™)) = xX™ (54) 


where Px is the projection on the nonnegative orthant, S$” is the (diagonal) scaling 
matrix, and o, is a suitably chosen step length [10]. Then iteration XT is 
obtained by a line search along the descent direction based on the Armijo rule. We 
can add that the method can be easily extended to the case where the convex set of 
the admissible solutions is defined by box and equality constraints. 

In the case of the minimization of the KL divergence, the diagonal scaling matrix 
is that suggested by RL, ie., $“” = X (with the addition of suitable upper and 
lower bounds), and the method shows the semi-convergence property as RL, but 
requires a much smaller number of iterations for obtaining a sensible reconstruction. 
In an application to the deconvolution of astronomical images [46], it has been 
shown that, even if the computational cost per iteration is about 30 % greater than 
that of RL, thanks to the reduction of the number of iterations it is possible to obtain 
a speedup, with respect to RL, ranging from 4 to more than 30, depending on the 
astronomical source and the noise level. Moreover, implementation on graphics 
processing units (GPU) allows to deconvolve a 2,048 x 2,048 image in a few 
seconds. 

Several iterative methods, modeled on RL, have been introduced for 
computing MAP estimates corresponding to different kinds of priors. A recent 
account can be found in [4]. A nice feature of SGP, whose convergence 
is proved in [10], is that it can be easily applied to this problem, ie., to 
the minimization of the function of Eq.50 (again, with the addition of box 
and equality constraints). The scaling is taken from the split-gradient method 
(SGM), proposed in [31], since this scaling is always nonnegative, while the 
scaling proposed in [26] may take negative values. In general the choice of 
the scaling is as follows. If the gradient of Jr(X) is split in the following 
way: 


— Vx IR(X) = UR(X) — Vr(X), (55) 
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where Ur, Vr are nonnegative arrays, then the scaling is given by the 
array 


xX 


OTF TR | me 


Of course SGP can be applied if Jr(X) is differentiable, and therefore, it can 
cover both smoothing regularization as given by Tikhonov and also edge-preserving 
regularization as given by smoothed TV (total variation) [12,65]. Finally, a difficult 
problem in the case of regularized problems is the choice of the value of the regu- 
larization parameter. An attempt in the direction of solving this problem is provided 
by a recently proposed discrepancy principle for Poisson data deconvolution [5]. 


Blind Deconvolution 


Blind deconvolution is the problem of image deblurring when the blur is unknown. 
In the case of a space-invariant model, the naive problem formulation is to solve 
the problem Y = H * X where only Y is known, where * denotes convolution. 
It is obvious that this problem is extremely undetermined and that there is an 
infinite set of pairs solving the equation. Among them also is the trivial solution 
X = Y, H = 6, where 6 denotes the usual delta function. Therefore, the problem 
must be formulated by introducing as far as possible all available constraints on both 
the object X and the PSF H. 

In the case of Poisson noise, several iterative methods have been proposed, which 
consist of alternating updates of the object and PSF by means of RL iterations or 
accelerated RL iterations. For instance, in [27] one RL iteration is used both on the 
object and the PSF. This algorithm was investigated, in the context of nonnegative 
matrix factorization (NMF), by Lee and Seung [32], but their convergence proof 
is incomplete, since only the monotonic decrease of the objective function is 
shown while, for a general descent method to be convergent, strongest Armijo-like 
decreasing conditions have to be verified. In general, the proposed approaches to 
blind deconvolution with Poisson data could be classified as methods of inexact 
alternating minimization applied to the KL divergence, as a function of both object 
and PSF. 

In a recent paper [9], in the context of NMF, convergence of inexact alternating 
minimization is proved if the iterative algorithm used for the inner iterations satisfies 
suitable conditions, which are satisfied by SGP. Therefore, this approach looks very 
suitable for the problem of blind deconvolution with Poisson data. The approach is 
applied in [47] using constraints both on the object and the PSF. The method applies 
to the imaging by ground-based telescopes since, as suggested in [20], one of the 
constraints on the PSF is provided by the Strehl ratio (SR; see Sect. 2), a parameter 
measuring the quality of the AO correction. Indeed we recall that the advantage of 
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SGP is not only fast convergence, if a suitable scaling of the gradient is used, but also 
the possibility of introducing suitable box and equality constraints on the solution. 
In particular the SR constraint excludes the trivial solution mentioned above. The 
method works well in the case of star systems. 


Deconvolution with a Sparsity Prior 


Another approach is to use the sparsity to model the data. A sparse model can 
be interpreted from a Bayesian standpoint, by assuming the coefficients a of the 
solution in the dictionary ® follow a leptokurtic PDF with heavy tails such as the 
generalized Gaussian distribution form: 


K 
pdf, (or,...,a) « PJ exp (—a ea) O<p<2. (57) 
k=1 


Between all possible solutions, we want the one which has the sparsest represen- 
tation in the dictionary ®. Putting together the log-likelihood function in the case 
of Gaussian noise o and the priors on a, the MAP estimator leads to the following 
optimization problem: 


K 
a al ; 
min =~ || — al +A) |lacllS 0 < p <2. (58) 


Q1,...,0K ea 


The sparsity can be measured through the ||@||9 norm (i.e., p = 0). This counts 
in fact the number of nonzero elements in the sequence. It was also proposed to 
convexify the constraint by substituting the convex ||q||; norm for the ||a||o norm 
[13]. Depending on the H operator, there are several ways to obtain the solution of 
this equation. 

A first iterative thresholding deconvolution method was proposed in [51] which 
consists of the following iterative scheme: 


XO+D = Py (X 4 HT (WDenyw (Y — HX))) (59) 


where P is the projection on the cone of nonnegative images and WDen is an 
operator which performs a wavelet thresholding, i.e., applies the wavelet transform 
of the residual R™ (i.e., R™ = Y — HX), thresholds some wavelet coefficients, 
and applies the inverse wavelet transform. Only coefficients that belong to the 
multiresolution support M“ [51] are kept, while the others are set to zero. At each 
iteration, the multiresolution support M“”) is updated by selecting new coefficients 
in the wavelet transform of the residual which have an absolute value larger than 
a given threshold. The threshold is automatically derived assuming a given noise 
distribution such as Gaussian or Poisson noise. 
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More recently, it was shown [14, 16, 24] that a solution of Eq. 58 for p = 1 can 
be obtained through a thresholded Landweber iteration: 


XT) = P, (WDen, (X” +H" (Y — HX™))) , (60) 


with ||H|| = 1. In the framework of monotone operator splitting theory, it was 
shown that for frame dictionaries, a slight modification of this algorithm converges 
to the solution [14]. Extension to constrained nonlinear deconvolution is proposed 
in [22]. 


Constraints in the Object or Image Domains 

Let us define the object domain O as the space in which the solution belongs and 
the image domain T as the space in which the observed data belongs (i.e., if X €¢ O 
then HX ¢€ Z). The constraint in (59) was applied in the image domain, while in 
(60) we have considered constraints on the solution. Hence, two different wavelet- 
based strategies can be chosen in order to regularize the deconvolution problem. 
The constraint in the image domain through the multiresolution support leads to a 
very robust way to control the noise. Indeed, whatever the nature of the noise, we 
can always derive robust detection levels in the wavelet space and determine scales 
and positions of the important coefficients. A drawback of the image constraints is 
that there is no guarantee that the solution is free of artifacts such as ringing around 
point sources. A second drawback is that image constraints can be used only if the 
point spread function is relatively compact, i.e., does not smear the information over 
the whole image. 

The property of introducing robust noise modeling is lost when applying the 
constraint in the object domain. For example, in the case of Poisson noise, there is no 
way (except using time-consuming Monte Carlo techniques) to estimate the level of 
the noise in the solution and to adjust properly the thresholds. The second problem 
with this approach is that, in fact, we try to solve two problems simultaneously 
(noise amplification and artifact control in the solution) with one parameter (i.e., 
4). The choice of this parameter is crucial, while such a parameter is implicit when 
using the multiresolution support. 

Ideally, constraints should be added in both the object and image domains in 
order to better control the noise by using the multiresolution support and avoid such 
a ringing artifact. 


Example 

A simulated Hubble Space Telescope Wide Field Camera image of a distant cluster 
of galaxies is shown in Fig. 14, upper left. The simulated data are shown in 
Fig. 14, upper right. The Richardson—Lucy and the wavelet solutions are shown, 
respectively, in Fig. 14, lower left and right. The Richardson—Lucy method amplifies 
the noise, which implies that the faintest objects disappear in the deconvolved 
image, while the wavelet starlet solution is stable for any kind of PSF, and any 
kind of noise modeling can be considered. 
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Fig. 14 Simulated Hubble Space Telescope Wide Field Camera image of a distant cluster of 
galaxies. Upper left: original, unaberrated, and noise-free. Upper right: input, aberrated, noise 
added. Lower left: restoration, Richardson—Lucy. Lower right, restoration starlet deconvolution 


Detection and Deconvolution 


The PSF is not needed with MVM. This is an advantage when the PSF is unknown 
or difficult to estimate, which happens relatively often when it is space variant. 
However, when the PSF is well determined, it becomes a drawback because known 
information is not used for the object reconstruction. This can lead to systematic 
errors in the photometry, which depends on the PSF and on the source signal-to- 
noise ratio. In order to preempt such a bias, a kind of calibration must be performed 
using simulations [50]. This section shows how the PSF can be used in the MVM, 
leading to a deconvolution. 


Object Reconstruction Using the PSF 


A reconstructed and deconvolved object X; can be obtained by searching for a 
signal X; such that the wavelet coefficients of HX; are the same as those of the 
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detected structures a;. If W describes the wavelet transform operator and P,, the 
projection operator in the subspace of the detected coefficients, the solution is found 
by minimization of 


min || a — Py (WHX) ||? (61) 


where a; represents the detected wavelet coefficients of the object O; and H is the 
PSE In this approach, each object is deconvolved separately. The flux related to the 
extent of the PSF will be taken into account. For point sources, the solution will 
be close to that obtained by PSF fitting. This problem is also different from global 
deconvolution in the sense that it is well constrained. Except for the positivity of the 
solution which is always true and must be used, no other constraint is needed. This 
is due to the fact that the reconstruction is performed from a small set of wavelet 
coefficients (those above a detection limit). The number of objects is the same as 
those obtained by the MVM, but the photometry and the morphology are different. 
The astrometry may also be affected. 


The Algorithm 


Any minimizing method can be used to obtain the solution X;. Since there is 
no problem of convergence, noise amplification, or ringing effect, the Van Cittert 
method was proposed on the grounds of its simplicity [57]. It leads to the following 
iterative scheme: 


xD ¥M4R (a: —P, (wHx;”)) (62) 


where R is the inverse wavelet transform, and the algorithm is: 


1. Setn to 0. 

2. Find the initial estimation X i by applying an inverse wavelet transform to the 
set a; corresponding to the detected wavelet coefficients in the data. 

3. Convolve X{"” with the PSF H: Y{”) = HX}. 


4. Determine the wavelet transform et z) a ye, 


5. Threshold all wavelet coefficients in (i ) at position and scales where nothing 
y”) 
has been detected (i.e., P,, operator). We get ol 


«) 
y, 
. Determine the residual a, = a; — a ) 


. Reconstruct the residual image R“ by applying an inverse wavelet transform. 
. Add the residual to the solution: X"F? = x + R™, 

. Threshold negative values in X oot) 

10. If o(R™)/o0(X) <e¢,thenn =n + 1 and go to step 3. 

11. X ate contains the deconvolved reconstructed object. 


Oo OND 
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In practice, convergence is very fast (less than 20 iterations). The reconstructed 
image (not deconvolved) can also be obtained just by reconvolving the solution with 
the PSF. 


Space-Variant PSF 


Deconvolution methods generally do not take into account the case of a space- 
variant PSF. The standard approach when the PSF varies is to decompose the image 
into blocks and to consider the PSF constant inside a given block. Blocks which 
are too small lead to a problem of computation time (the FFT cannot be used), 
while blocks which are too large introduce errors due to the use of an incorrect PSF. 
Blocking artifacts may also appear. Combining source detection and deconvolution 
opens up an elegant way for deconvolution with a space-variant PSF. Indeed, a 
straightforward method is derived by just replacing the constant PSF at step 3 of 
the algorithm with the PSF at the center of the object. This means that it is not the 
image which is deconvolved, but its constituent objects. 


Undersampled Point Spread Function 


If the PSF is undersampled, it can be used in the same way, but results may not be 
optimal due to the fact that the sampled PSF varies depending on the position of the 
source. If an oversampled PSF is available, resulting from theoretical calculation or 
from a set of observations, it should be used to improve the solution. In this case, 
each reconstructed object will be oversampled. Equation 61 must be replaced by 


min || oj — Py (WD) HX)) |? (63) 


where PD; is the averaging-decimation operator, consisting of averaging the data in 
the window of size / x / and keeping only one average pixel for each / x / block. 


Example: Application to Abell 1689 ISOCAM Data 


Figure 15 (left) shows the detections (isophotes) obtained using the MVM method 
without deconvolution on ISOCAM data. The data were collected using the 6 arcsec 
lens at 6.75 jum. This was a raster observation with 10s integration time, 16 raster 
positions, and 25 frames per raster position. The noise is nonstationary, and the 
detection of the significant wavelet coefficients was carried out using the root mean 
square error map R(x, y) by the method described in [50]. The isophotes are 
overplotted on an optical image (NTT, band V) in order to identify the infrared 
source. Figure 15 (right) shows the same treatment but using the MVM method 
with deconvolution. The objects are the same, but the photometry is improved, and 
it is clearly easier to identify the optical counterpart of the infrared sources. 
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Fig. 15 Abell 1689: left, ISOCAM source detection (isophotes) overplotted on an optical image 
(NTT, band V). The ISOCAM image is a raster observation at 71m. Right, ISOCAM source 
detection using the PSF (isophotes) overplotted on the optical image. Compared to the left panel, 
it is clearly easier to identify the detected infrared sources in the optical image 


6 Conclusion 


In this chapter we have used the sparsity principle that now occupies a very central 
role in signal processing. We have discussed the vision models within which the 
sparsity principle is applied. Finally, we have reviewed the use of the starlet wavelet 
transform as a prime technique in order to apply the sparsity principle in the 
context of vision models in various application domains. Among the latter are object 
detection coupled with denoising, deconvolution and ltering generally. Issues of 
algorithmic optimization and of statistical modeling entered into our discussion 
on various occasions. Many examples and case studies were used to demonstrate 
the powerfulness of the approaches described for astronomical data analysis and 
processing. 


Cross-References 


> Energy Minimization Methods 

> Iterative Solution Methods 

> Large-Scale Inverse Problems in Imaging 

> Linear Inverse Problems 

> Numerical Methods and Applications in Total Variation Image Restoration 
> Regularization Methods for Ill-Posed Problems 

> Splines and Multiresolution Analysis 

> Statistical Methods in Imaging 

> Total Variation in Imaging 
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Abstract 


Images in scientific visualization are the end product of data processing. Starting 
from higher-dimensional data sets such as scalar, vector, and tensor fields given 
on 2D, 3D, and 4D domains, the objective is to reduce this complexity to 
two-dimensional images comprehensible to the human visual system. Various 
mathematical fields such as in particular differential geometry, topology (theory 
of discretized manifolds), differential topology, linear algebra, Geometric Alge- 
bra, vector field and tensor analysis, and partial differential equations contribute 
to the data filtering and transformation algorithms used in scientific visualization. 
The application of differential methods is core to all these fields. The following 
chapter will provide examples from current research on the application of these 
mathematical domains to scientific visualization. Ultimately the use of these 
methods allows for a systematic approach for image generation resulting from 
the analysis of multidimensional datasets. 


1 Introduction 


Scientists need an alternative to numbers. The use of images is a technical reality nowadays 
and tomorrow it will be an essential requisite for knowledge. The ability of scientists to 
visualize calculations and complex simulations is absolutely essential to ensure the integrity 
of analyses, to promote scrutiny in depth and to communicate the result of such scrutiny to 
others... The purpose of scientific calculation is looking, not enumerating. It is estimated 
that 50 % of the brain’s neurons are associated with vision. Visualization in a scientific 
calculation is aimed at putting this neurological machinery to work [56]. 


Since this visionary quote from an article in 1987, scientific visualization, benefiting 
from the affordable graphics hardware driven by the computer gaming industry, has 
grown rapidly. Beyond academic research interests it has become also a consumer 
market with practical applicability in industry and medicine. Still there are yet many 
gaps that are left open due to the unequal evolution velocities in different fields. 
Once, there is the human mind that is not able to keep up with the deluge of visual 
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information which can be produced with modern technology. Many scientists still 
prefer looking at numbers instead of utilizing modern display technology. At the 
same time, data can be produced by modern supercomputers that is far beyond the 
ability of even high-end graphics engines to be processed. Data sets originating 
from numerical simulations of physical processes will usually be three-dimensional 
or four-dimensional, with images just the final result of the process of scientific 
visualization. In this context images are the means to analyze data set of higher 
dimensions. 

Reducing numerical data sets to images is known as the concept of the visual- 
ization pipeline. In its simplest form it consists of a data source (m-dimensional), 
a data filter (an algorithmic operation), and a data sink (an image). Data filters 
need to understand the structure and meaning of the multidimensional input data 
and to operate efficiently on them. This involves various mathematical fields such 
as in particular differential geometry, topology (theory of discretized manifolds), 
differential topology, linear algebra, Geometric Algebra, vector field and tensor 
analysis, and partial differential equations. Within a scientific visualization process, 
all these mathematical fields will work together, with more or less weighting. 
We subsume this set of mathematical domains as “differential methods” in this 
chapter as the concept of differentiation is fundamental to their approach of data 
analysis. The following sections will demonstrate the application of the respective 
mathematical fields to visual analysis by virtue of examples of ongoing research. 

In Sect.2 we discuss the general issue of how to lay out data to model the 
structure of space and time, as we know it from mathematics as foundation for 
further operations. Frequently visualization algorithms are implemented ad hoc, 
given the problem, inventing the solution with highest performance. This allegedly 
reasonable approach comes with an unfortunate downside: incompatibility among 
independently developed solutions, which impacts data exchange and interfacing 
complementary implementations. However, when keeping a common data model in 
mind right from the earliest steps of conceiving some algorithm, interoperability can 
be achieved at no cost with same performance as solitary solutions. 

Given a solid foundation for data structures, Sect.3 demonstrates how to 
formulate differential operators using the concepts of chains, cochains, homology, 
and cohomology. Since in computer graphics and visualization we have to deal with 
discretized spaces, we arrive in the mathematical field of topology, as an essential 
descriptive tool for meshes and all nontrivial grid structures. 

When considering mathematics as a language unifying computer science, we 
need to even more think about a common denominator within mathematics itself. 
Geometric Algebra is a relatively new — or, rather, rediscovered — branch of 
mathematics that is very promising. It is extraordinarily visually intuitive while 
covering the abstractions of Clifford algebra as used in quantum mechanics equally 
well as the formulations of curved space in general relativity. However, even 
independent of such physics-oriented applications, Geometric Algebra has found its 
merits within computer graphics itself. Section 4 will talk about the elegant usage 
of five-dimensional projective conformal Geometric Algebra to handle primitives in 
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computer graphics and eventually implement the ray-tracing algorithm with a few, 
well-defined algebraic operations. 

The general goal of visualization is to give insight into large and complex data 
sets. Due to the sheer size of the data sets alone, it is favorable if not necessary 
to automate at least parts of the analysis. A way to achieve this is by extracting 
features. Features can either be certain quantities derived from a data set or a 
mathematically well-defined, geometric object (point, line, surface, ...) with its 
definition and interpretation depending on the underlying application, but usually 
it represents important structures (e.g., vortex, stagnation point) or changes to such 
structures (events, bifurcations). A feature-based visualization aims at the reduction 
of information to guide a user to the most interesting parts of a data set. In Sect. 5 we 
describe some important approaches to feature-based visualization of vector fields. 
These include investigation of derived quantities such as vortices (section “Derived 
Measures of Vector Fields”) and the topology of vector fields (section “Topology of 
Vector Fields”). These approaches have become a standard tool for the analysis of 
vector fields. 

Finally, in Sect. 6 we explore the capabilities of partial differential equations for 
the filtering and regularization of image data sets. Applications are enhancing image 
quality by reducing noise or similar artifacts, as well as the visualization of vector 
and tensor fields. 


2 Modeling Data via Fiber Bundles 


Purely numerical algorithms in C++ can be abstracted from concrete data struc- 
tures using programming techniques such as generic programming [79]. However, 
generic algorithms still need to make certain assumptions about the data they 
operate on. The question remains what these concepts are that describe “data”: what 
properties should be expected by some algorithm from any kind of data provided for 
scientific visualization? Moreover, consistency among concepts shared by indepen- 
dent algorithms is also required to achieve interoperability among algorithms and 
eventually (independently developed) applications. While any particular problem 
can be addressed by some particular solution, a common concept allows to build a 
framework instead of just a collection of tools. Tools are what an end user needs 
to solve a particular problem with a known solution. However, when a problem is 
not yet clearly defined and a solution unknown, then a framework is required that 
allows exploration of various approaches and eventually adaption toward a specific 
direction that does not exist a priori. 

The concept of how to lay out data to perform visualization operations in a 
common framework constitutes a data model for visualization. Many visualization 
applications are to a greater or lesser extent a collection of tools, even when bundled 
together within the same software library or binary. Consequently, interoperability 
between different applications and their corresponding file formats is hard or 
impossible. Only very few implementations adhere to the vision of a common data 
model across the various data types for visualization. The idea of a common data 
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model is frequently undervalued or even disregarded as being impossible. However, 
as D. Butler said, “The proper abstractions for scientific data are known. We just 
have to use them” [16]. 

D. Butler was following the mathematical concepts of fiber bundles [16], or 
more specific, vector bundles [15], to model data. The IBM Data Explorer, one 
of the earliest visualization applications, now open source and known as “OpenDX 
(http://www.opendx.org),” implemented this concept successfully [76]. These ideas 
have been revived and expanded by [7] leading to a hierarchical data structure 
consisting of a noncyclic graph in seven levels. It can be seen as largely keyword- 
free, hierarchical version of the OpenDX model, seeking to cast the information 
and relationships provided in original model into a grouping structure. This data 
model will be reviewed in the following, together with its mathematical background. 
Section “Differential Geometry: Manifolds, Tangential Spaces, and Vector Spaces” 
will review the basic mathematical structures that are used to describe space and 
time. Section “Topology: Discretized Manifolds” will introduce the mathematical 
formulation of discretized space. Based on this background, section “Ontological 
Scheme and Seven-Level Hierarchy” will present a scheme that is able to cover the 
described mathematical structures. 


Differential Geometry: Manifolds, Tangential Spaces, and Vector 
Spaces 


Space and time in physics is modeled via the concept of a differentiable manifold. 
As scientific visualization deals with data given in space and time, following these 
concepts is reasonable. In short, a manifold is a topological space that is locally 
homeomorphic to R”. However, not all data occurring in scientific visualization 
are manifolds. The more general case of topological spaces will be discussed in 
sections “Topology: Discretized Manifolds” and “Topology.” 

A vector space over a field F(such as R) is a set V together with two binary 
operations vector addition +: VxV — V and scalar multiplicationo: FxV > V. 
The mathematical concept of a vector is defined as an element v € V. A vector 
space is closed under the operations + and 9, i.e., for all elements u,v € V and all 
elements A € F there isu+v € V andd ou € V (vector space axioms). The vector 
space axioms allow computing the differences of vectors and therefore defining the 
derivative of a vector-valued function v(s) : R > V as 


v(s + ds) — v(s) 


ds (1) 


a (s) := li 
a := limgs-so 


A manifold in general is not a vector space. However, a differentiable manifold M@ 
allows to define a tangential space Tp (MM) at each point P which has vector space 
properties. 
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Tangential Vectors 
In differential geometry, a tangential vector on a manifold M is the operator £ that 
computes the derivative along a curve g(s) : R > M for an arbitrary scalar-valued 
function f: M—-R: 


4 yf = a0) os 
s* |q(s) ds 

Tangential vectors fulfill the vector space axioms and can therefore be expressed as 
linear combinations of derivatives along the n coordinate functions x“: M—> R 
with 44 = 0...n — 1, which define a basis of the tangential space 77(s)(M) on the 
n-dimensional manifold M at each point g(s) € M: 


‘fF ee i : f= af (3) 


where {q}" are the components of the tangential vector i in the chart {x} and 
{0,,} are the basis vectors of the tangential space in this chart. In the following 
text the Einstein sum convention is used, which assumes implicit summation over 
indices occurring on the same side of an equation. Often tangential vectors are used 
synonymous with the term “vectors” in computer graphics when a direction vector 
from point A to point B is meant. A tangential vector on an n-dimensional manifold 
is represented by n numbers in a chart. 


Covectors 

The set of operations df: T(M) — R that map tangential vectors v € T(M) toa 
scalar value v(f) for any function f : M — R defines another vector space which 
is dual to the tangential vectors. Its elements are called covectors: 


a) 
<dfiv >= df) =f) = inf = oo (4) 
Covectors fulfill the vector space axioms and can be written as linear combination 
of covector basis functions dx": 


af =: Sat (5) 
whereby the dual basis vectors fulfill the duality relation 
Sa asa (6) 
ial ‘ xv: 0 


The space of covectors is called the cotangential space Tp*(M). A covector on an 
n-dimensional manifold is represented by m numbers in a chart, same as a tangential 
vector. However, covectors transform inverse to tangential vectors when changing 
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Fig. 1 The (trivial) constant vector field along the z-axis viewed as vector field 0, and as covector 
field dz. (a) Vector field 0,. (b) Duality relationship among 0, and dz. (ce) Co-vector field dz 


coordinate systems, as is directly obvious from Eq. (6) in the one-dimensional case: 
as < dx°,d9 >= 1 must be sustained under coordinate transformation, dx° must 
shrink by the same amount as 09 grows when another coordinate scale is used 
to represent these vectors. In higher dimensions this is expressed by an inverse 
transformation matrix. 

In Euclidean three-dimensional space, a plane is equivalently described by a 
“normal vector,’ which is orthogonal to the plane. While “normal vectors” are 
frequently symbolized by an arrow, similar to tangential vectors, they are not the 
same, rather they are dual to tangential vectors. It is more appropriate to visually 
symbolize them as a plane. This visual is also supported by (5), which can be 
interpreted as the total differential of a function f: a covector describes the change 
of a function f along a direction as specified by a tangential vector v. A covector 
V can thus be visually imagined as a sequence of coplanar (locally flat) planes 
at distances given by the magnitude of the covector that count the number of 
planes which are crossed by a vector w. This number is V(w). For instance, for 
the Cartesian coordinate function x, the covector dx “measures” the “crossing rate” 
of a vector w in the direction along the coordinate line x; see Figs. 1 and 2. On 
an n-dimensional manifold a covector is correspondingly symbolized by a (n — 1)- 
dimensional subspace. 


Tensors 
A tensor T”" of rank n x m is a multi-linear map of n vectors and m covectors to a 
scalar 


T™ : T(M)x...T(M), x T*(M) x...T*(M)m > R (7) 


Tensors are elements of a vector space themselves and form the tensor algebra. 
They are represented relative to a coordinate system by a set of k”*” numbers 
for a k-dimensional manifold. Tensors of rank 2 may be represented using matrix 
notation. Tensors of type 7; ° are equivalent to covectors and called co-variant; in 
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a b c 
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Fig. 2. The basis vector and covector fields induced by the polar coordinates {r, 3, #}. (a) Radial 
field dr 0,. (b) Azimuthal field dé dg view of the equatorial plane (z-axis towards eye). (c) Altitudal 
field d@ dg slice along the z-axis 


matrix notation (relative to a chart) they correspond to rows. Tensors of type To ! 
are equivalent to a tangential vector and are called contra-variant, corresponding to 
columns in matrix notation. The duality relationship between vectors and covectors 
then corresponds to the matrix multiplication of a 1 x 1 row with an x 1 column, 
yielding a single number 


b° 
b! 
<a,b >=< a" 0y,b,dx" > = (a°a'...a") (8) 


b” 


By virtue of the duality relationship (6), the contraction of lower and upper indices 
is defined as the interior product t of tensors, which reduces the dimensionality of 
the tensor: 


eT eT) TG a Hr ty (9) 


The interior product can be understood (visually) as a generalization of some 
“projection” of a tensor onto another one. 

Of special importance are symmetric tensors of rank two g € ry with g : 
T(M) x T(M) > R: u,v & g(u,v),g(u,v) = g(v,u), as they can be used 
to define a metric or inner product on the tangential vectors. Its inverse, defined by 
operating on the covectors, is called the co-metric. A metric, same as the co-metric, 
is represented as a symmetric n <n matrix in a chart for an n-dimensional manifold. 

Given a metric tensor, one can define equivalence relationships between tangen- 
tial vectors and covectors, which allow to map one into each other. These maps are 
called the “musical isomorphisms,” b and {f, as they raise or lower an index in the 
coordinate representation: 
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b: T(M)>T*(M): vYO, v4 g,,dx? (10) 
tt: T*(M)>T(M): V,dx" > Vi ga, (11) 


As an example application, the “gradient” of a scalar function is given by Vf = 
ttdf using this notation. In Euclidean space, the metric is represented by the identity 
matrix and the components of vectors are identical to the components of covectors. 
As computer graphics usually is considered in Euclidean space, this justifies the 
usual negligence of distinction among vectors and covectors; consequently graphics 
software only knows about one type of vectors which is uniquely identified by its 
number of components. However, when dealing with coordinate transformations or 
curvilinear mesh types, distinguishing between tangential vectors and covectors is 
unavoidable. Treating them both as the same type within a computer program leads 
to confusions and is not safe. 


Exterior Product 

The exterior product \: V x V — A(V) is an algebraic construction generating 
vector space elements of higher dimensions from elements of a vector space V. 
The new vector space is denoted A(V). It is alternating, fulfilling the property 
vAu = -—u Av Vu,v € V (which results in v Av = 0 Vv € V). 
The exterior product defines an algebra on its elements, the exterior algebra 
(or Grassmann algebra). It is a sub-algebra of the tensor algebra consisting of 
the antisymmetric tensors. The exterior algebra is defined intrinsically by the 
vector space and does not require a metric. For a given n — dimensional vector 
space V, there can at most be nth power of an exterior product, consisting of 
n different basis vectors. The (7 + 1)th power must vanish, because at least 
one basis vector would occur twice, and there is exactly one basis vector in 
A"(V). 

Elements v € A‘(V) are called k-vectors, whereby two-vectors are also called 
bi-vectors and three-vectors tri-vectors. The number of components of a k-vector 
of an n-dimensional vector space is given by the binomial coefficient {n}{k}. For 
n = 2 there are two one-vectors and one bi-vector, form = 3 there are three one- 
vectors, three bi-vectors, and one tri-vector. These relationships are depicted by the 
Pascal’s triangle, with the row representing the dimensionality of the underlying 
base space and the column the vector type: 


1 2 1 (12) 
1 3 3 1 
14 6 4 1 


As can be easily read off, for a four-dimensional vector space, there will be four 
one-vectors, six bi-vectors, four tri-vectors, and one four-vector. The n-vector of 
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an n-dimensional vector space is also called a pseudoscalar, the (n — 1) vector a 
pseudo-vector. 


Visualizing Exterior Products 
An exterior algebra is defined on both the tangential vectors and covectors on a 
manifold. A bi-vector v formed from tangential vectors is written in chart as 


v=ved, A dy (13) 
and a bi-covector U formed from covectors is written in chart as 
U = U,y,dx" A dx” (14) 


They both have {n}{2} independent components, due to v*” = —v"* and Uyy = 
—U,,, (three components in 3D, six components in 4D). A bi-tangential vector (13) 
can be understood visually as an (oriented, i.e., signed) plane that is spun by the two 
defining tangential vectors, independently of the dimensionality of the underlying 
base space. A bi-covector (14) corresponds to the subspace of an n-dimensional 
hyperspace where a plane is “cut out.” In three dimensions these visualizations 
overlap: both a bi-tangential vector and a covector correspond to a plane, and 
both a tangential vector and a bi-covector correspond to one-dimensional direction 
(“arrow”). In four dimensions, these visuals are more distinct but still overlap: a 
covector corresponds to a three-dimensional volume, but a bi-tangential vector is 
represented by a plane same as a bi-covector, since cutting out a 2D plane from 
four-dimensional space yields a 2D plane again. Only in higher dimensions these 
symbolic representations become unique. However, both a co-vector and a pseudo- 
vector will always correspond to (1.e., appear as) an (7 — 1)-dimensional hyperspace. 


Vidx" > Vavay..n—19a9 \ Oa, A+++ Day —1 (15) 
VY On > Vege tn dx A dx A... dx! (16) 


A tangential vector — lhs of (16) — can be understood as one specific direction. 
Equivalently, it can be seen as “cutting off” all but one (n — 1)-dimensional 
hyperspaces from the full n-dimensional space. This equivalence is expressed via 
the interior product of a tangential vector v with a pseudo-co-scalar Q2 yielding a 
pseudo-covector V (17). Similarly, the interior product of a pseudo-vector with a 
pseudo-co-scalar yields a tangential vector (17): 


tg: T(M) > (T*)"-9)(M) : VB tev (17) 


tai T’-)(M) > T*(M): VB tgy (18) 
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Pseudoscalars and pseudo-co-scalars will always be scalar multiples of the basis 
vectors dao A a1 A ..-Oun and dxj A dxf A ...dxt. However, when inversing a 
coordinate x“ — —x*, they flip sign, whereas a “true” scalar does not. An example 
known from Euclidean vector algebra is the allegedly scalar value constructed from 
the dot and cross product of three vectors V(u,v,w) = u-(v x w) which is the 
negative of when its arguments are flipped: 


V(u, v, w) = —V(—u, —v, —w) = —u-(—v x —w) (19) 
which is actually more obvious when (19) is written as exterior product: 
Viu,v,w) =uAvAw =Vd0A 01 A 02 (20) 


The result (20) actually describes the multiple of a volume element span by the basis 
tangential vectors 0,, — any volume must be a scalar multiple of this basis volume 
element but can flip sign if another convention on the basis vectors is used. This 
convention depends on the choice of a right-handed versus left-handed coordinate 
system and is expressed by the orientation tensor 2 = +0 A 0) A 02. In computer 
graphics, both left-handed and right-handed coordinate systems occur, which may 
lead to lots of confusions. 

By combining (18) and (11) — requiring a metric — we get a map from pseudo- 
vectors to vectors and reverse. This map is known as the Hodge star operator “*”: 


21 T°’ -)(M) > T(M): VE = thaV (21) 


The same operation can be applied to the covectors accordingly and generalized 
to all vector elements of the exterior algebra on a vector space, establishing a 
correspondence between k — vectors and n — k-vectors. The Hodge star operator 
allows to identify vectors and pseudo-vectors, similar to how a metric allows to 
identify vectors and covectors. The Hodge star operator requires a metric and an 
orientation (2. 

A prominent application in physics using the hodge star operator are the Maxwell 
equations, which, when written based on the four-dimensional potential A = 
Vodx® + Apdx* (Vo the electrostatic, A, the magnetic vector potential), take the 
form 


d,dA=J (22) 


with J the electric current and magnetic flow, which is zero in vacuum. The 


combination d * d is equivalent to the Laplace operator “U1,” which indicates 
that (22) describes electromagnetic waves in vacuum. 


Geometric Algebra 
Geometric Algebra is motivated by the intention to find a closed algebra on a 
vector space with respect to multiplication, which includes existence of an inverse 
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operation. There is no concept of dividing vectors in “standard” vector algebra. 
Neither the inner or outer product has provided vectors of the same dimensionality 
as their arguments, so they do not provide a closed algebra on the vector space. 

Geometric Algebra postulates a product on elements of a vector space u, v, w € V 
that is associative (uv)w = u(vw), left distributive u(v + w) = uv 4+ uw, and right 
distributive (u + v)w = uw + vw and reduces to the inner product as defined by the 
metric v? = g(v, v). It can be shown that the sum of the outer product and the inner 
product fulfill these requirements; this defines the geometric product as the sum of 
both: 


uv:=uAv+u-v (23) 


Since uAv and u-v are of different dimensionalities ({7} {{2} and {n} { {0}, respec- 
tively), the result must be in a higher-dimensional vector space of dimensionality 
{n} {{2} + {n} {{0}. This space is formed by the linear combination of k-vectors; 
its elements are called multivectors. Its dimensionality is pa . ) = 2", 


k 
For instance, in two dimensions, the dimension of the space of multivectors is 
2? = 4. A multivector V, constructed from tangential vectors on a two-dimensional 
manifold, is written as 


V =V°+ V'0o + :V70; + : V2 A Oy (24) 


with V“ the four components of the multivector in a chart. For a three-dimensional 
manifold, a multivector on its tangential space has 2? = 8 components and is written 
as 


V = V°+ 
V' 06 + V0, + V20o+ 
V400 Ad, + VO, A 02 + Vd, A 09+ 
V700 A 01 A 02 


(25) 


with V* the eight components of the multivector in a chart. The components of 
a multivector have a direct visual interpretation, which is one of the key features 
of Geometric Algebra. In 3D, a multivector is the sum of a scalar value, three 
directions, three planes, and one volume. These basis elements span the entire 
space of multivectors. Geometric Algebra provides intrinsic graphical insight to 
the algebraic operations. Its application for computer graphics will be discussed 
in Sect. 4. 


Vector and Fiber Bundles 

The concept of a fiber bundle data model is inspired by its mathematical correspon- 
dence. In short, a fiber bundle is a topological space that looks locally like a product 
space B x F of a base space B and a fiber space F’. 
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The fibers of a function f: X — Y are the pre-images or inverse images of the 
points y € Y, i.e., the sets of all elements x € X with f(x) = y: 


f'0) ={& € XIf@ =y} 


is a fiber of f (at the point y). A fiber can also be the empty set. The union set of all 
fibers of a function is called the total space. The definition of a fiber bundle makes 
use of a projection mappr,, which is a function that maps each element of a product 
space to the element of the first space: 


pniXxY 7 Xx 


(x,y) > x 


Let E, B be topological spaces and f: E — B a continuous map. (E£, B, f) is 
called a (fiber) bundle if there exists a space F’ such that the union of fibers of a 
neighborhood U, C B of each point b € B is homeomorphic to U; x F such that 
the projection pr; of Up x F is Up again: 


(E, B, f : E> B) bundle <> 4F : Vb € B: Uy: f—'(Up) Up x F 
and pr,|(U, x F) =U, 


E is called the total space E, B is called the base space, and f : E — B the 
Projection map. The space F is called the fiber type of the bundle or simply the 
fiber of the bundle. In other words, the total space can be written locally as a product 
space of the base space with some space F’. The notation F(B) = (E, B, f) will 
be used to denote a fiber bundle over the base space B. It is also said that the space 
F fibers over the base space B. 

An important case is the tangent bundle, which is the union of all tangent spaces 
T, (M) on a manifold M together with the manifold 7(M) := {(p,v) : p € 
M,v € T,(M)}. Every differentiable manifold possesses a tangent bundle 7 (M). 
The dimension of 7 (M) is twice the dimension of the underlying manifold M, its 
elements are points plus tangential vectors. T,, (M/) is the fiber of the tangent bundle 
over the point p. 

If a fiber bundle over a space B with fiber F can be written as B x F globally, 
then it is called a trivial bundle (B x F, B, pr,). In scientific visualization, usually 
only trivial bundles occur. A well-known example for a nontrivial fiber bundle is the 
MoObius strip. 


Topology: Discretized Manifolds 


For computational purposes, a topological space is modeled by a finite set of points. 
Such a set of points intrinsically carries a discrete topology by itself, but one usually 
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considers embeddings in a space that is homeomorphic to Euclidean space to define 
various structures describing their spatial relationships. 

A subset c C X of a Hausdorff space X is a k-cell if it is homeomorphic to 
an open k-dimensional ball in R”. The dimension of the cell is k. Zero-cells are 
called vertices, one-cells are edges, two-cells are faces or polygons, and three-cells 
are polyhedra — see also section “Chains.” An n-cell within an n-dimensional space 
is just called a “cell.” (n — 1)-cells are sometimes called “facets” and (n — 2)-cells 
are known as “ridges.” For k-cells of arbitrary dimension, incidence and adjacency 
relationships are defined as follows: two cells c;, cz are incident if c, C 0c2, where 
dc. denotes the border of the cell c2. Two cells of the same dimension can never 
be incident because dim(c,) # dim(c2) for two incident cells c,, cz. cy is a side 
of cp if dim(c,) < dim(c2), which may be written as c; < cz. The special case 
dim(c,) = dim(c2) — 1 may be denoted by c, ~ c2. Two k -cells c), cz with k > 0 
are called adjacent if they have a common side, i.e., 


cellc},co adjacent >4 cell f: f<a,f<o@ 


For k = 0, two zero-cells (i.e., vertices) vj, v2 are said to be adjacent if there 
exists a one-cell (edge) e which contains both, i.e., vy) < e and v2 < e. Incidence 
relationships form an incidence graph. A path within an incidence graph is a cell 
tuple: a cell-tuple C within an n-dimensional Hausdorff space is an ordered sequence 
of k-cells (Cy, Cn—1,---,€1,€0) Of decreasing dimensions such that VO < i < 
n : Cj-1 < c;. These relationships allow to determine topological neighborhoods: 
adjacent cells are called neighbors. The set of all k + 1 cells which are incident 
to a k-cell forms a neighborhood of the k-cell. The cells of a Hausdorff space 
X constitute a topological base, leading to the following definition: a (“closure- 
finite, weak-topology”) CW-complex C, also called a decomposition of a Hausdorff 
space X, is a hierarchical system of spaces XT) C X® cc XM CL. XM, 
constructed by pairwise disjoint open cells c C X with the Hausdorff topology 
Ucee", such that X) is obtained from X("'~) by attaching adjacent n-cells to each 
(n—1)-cell and X‘~) = Q. The respective subspaces X “”) are called the n-skeletons 
of X. A CW complex can be understood as a set of cells which are glued together 
at their subcells. It generalizes the concept of a graph by adding cells of dimension 
greater than 1. 

Up to now, the definition of a cell was just based on a homeomorphism of the 
underlying space X and R”. Note that a cell does not need to be “straight,” such that, 
e.g., a two-cell may be constructed from a single vertex and an edge connecting the 
vertex to itself, as, e.g., illustrated by J. Hart [34]. Alternative approaches toward 
the definition of cells are more restrictively based on isometry to Euclidean space, 
defining the notion of “convexity” first. However, it is recommendable to avoid the 
assumption of Euclidean space and treating the topological properties of a mesh 
purely based on its combinatorial relationships. 
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Ontological Scheme and Seven-Level Hierarchy 


The concept of the fiber bundle data model builds on the paradigm that numerical 
data sets occurring for scientific visualization can be formulated as trivial fiber 
bundles (see section “Vector and Fiber Bundles”). Hence, data sets may be 
distinguished by their properties in the base space and the fiber space. At each point 
of the — discretized — base space, there are some data in the fiber space attached. 
Basically a fiber bundle is a set of points with neighborhood information attached 
to each of them. An n-dimensional array is a very simple case of a fiber bundle with 
neighborhood information given implicitly. 

The structure of the base space is described as a CW complex, which categorizes 
the topological structure of an n-dimensional base space by a sequence of k- 
dimensional skeletons, with 0 < k <n. These skeletons carry certain properties of 
the data set: the zero-skeleton describes vertices, the one-skeleton refers to edges, 
two-skeleton to the faces, etc., of some mesh (a triangulation of the base space). 
Structured grids are triangulations with implicitly given topological properties. For 
instance, a regular n-dimensional grid is one where each point has 2” neighbors. 

The structure of the fiber space is (usually) not discrete and given by the 
properties of the geometrical object residing there, such as a scalar, vector, covector, 
and tensor. Same as the base space, the fiber space has a specific dimensionality, 
though the dimensionality of the base space and fiber space is independent. Figure 3 
demonstrates example images from scientific visualization classified via their fiber 
bundle structure. If the fiber space has vector space properties, then the fiber bundle 
is a vector bundle and vector operations can be performed on the fiber space, such 
as addition, multiplication, and derivation. 

The distinction between base space and fiber space is not common use in 
computer graphics, where topological properties (base space) are frequently inter- 
mixed with geometrical properties (coordinate representations). Operations in the 
fiber space can, however, be formulated independently from the base space, which 
leads to a more reusable design of software components. Coordinate information, 
formally part of the base space, can as well be considered as fiber, leading to further 
generalization. The data sets describing a fiber are ideally stored as contiguous 
arrays in memory or disk, which allows for optimized array and vector operations. 
Such a storage layout turns out to be particularly useful for communicating data 
with the GPU using vertex buffer objects: the base space is given by vertex 
arrays (e.g., OpenGL g/VertexPointer), and fibers are attribute arrays (e.g., OpenGL 
glVertexAttribPointer), in the notation of computer graphics. While the process of 
hardware rendering in its early times had been based on procedural descriptions 
(cached in display lists), vertex buffer objects are much faster in state-of-the-art 
technology. Efficient rendering routines are thus implemented as maps from fiber 
bundles in RAM to fiber bundles in GPU memory (eventually equipped with a GPU 
shader program). 

A complex data structure (such as some color-coded time-dependent geometry) 
will be built from many data arrays. The main question that needs to be answered by 
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Fig. 3 Fiber bundle classification scheme for visualization methods:dimensionality of the base 
space (involving the k-skeleton of the discretized manifold) and dimensionality of the fiber space 
(involving the number of field quantities per element, zero referring to display of the mere 
topological structure). (a) Zero-cells, OD. (b) Zero-cells, 1D. (c) Zero-cells, 3D. (d) Zero-cells, 
6D. (e) One-cells, OD. (f) One-cells, 1D. (g) One-cells, 3D. (h) One-cells, 6D. (i) Two-cells, OD. 
(j) Two-cells, 1D. (k) Two-cells, 3D. (1) Two-cells, 6D. Gm) Three-cells, 0D. (n) Three-cells, 1D. 
(0) Three-cells, 3D. (p) Three-cells, 6D 


a data model is how to assign a semantic meaning to each of these data arrays — what 
do the numerical values actually mean? It is always possible to introduce a set of 
keywords with semantics attached to them. In addition, the introduction of keywords 
also reduces the number of possible identifiers available for user-specific purpose. 
This problem is also known as “name space pollution”. The approach followed in the 
data model presented in [7] is to avoid use of keywords as much as possible. Instead, 
it assigns the semantics of an element of the data structure into the placement of 
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this element. The objective is to describe all data types that occur in an algorithm 
(including file reader and rendering routines) within this model. It is formulated as 
a graph of up to seven levels (two of them optional). Each level represents a certain 
property of the entire data set, the Bundle. These levels are called: 


. Slice 

Grid 

. Skeleton 

. Representation 

. Field 

. (Fragment) 

. (Compound Elements) 


Actual data arrays are stored only below the “Field” level. Given one hierarchy level, 
the next one is accessed via some identifier. The type of this identifier differs for 
each level: numerical values within a Skeleton level are grouped into Representation 
objects, which hold all information that is relative to a certain “representer.’ Such 
a representer may be a coordinate object that, for instance, refers to some Cartesian 
or polar chart, or it may well be another Skeleton object, either within the same 
Grid object or even within another one. An actual data set is described through the 
existence of entries in each level. Only two of these hierarchy levels are exposed 
to the end user; these are the Grid and Field levels. Their corresponding textual 
identifiers are arbitrary names specified by the user. 


Hierarchy object Identifier type Identifier semantic 
Bundle Floating point number Time value 

Slice String Grid name 

Grid Integer set Topological properties 
Skeleton Reference Relationship map 
Representation String Field name 

Field Multidimensional index Array index 


A Grid is subset of data within the Bundle that refers to a specific geometrical 
entity. A Grid might be a mesh carrying data such as a triangular surface, a 
data cube, a set of data blocks from a parallel computation, or many other data 
types. A Field is the collection of data sets given as numbers on a specific 
topological component of a Grid, for instance, floating point values describing 
pressure or temperature on a Grid’s vertices. All other levels of the data model 
describe the properties of the Bundle as construction blocks. The usage of these 
construction blocks constitutes a certain language to describe data sets. A Slice 
is identified by a single floating point number representing time (generalization to 
arbitrary-dimensional parameter spaces is possible). A Skeleton is identified by its 
dimensionality, index depth (relationship to the vertices of a Grid), and refinement 
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Refinement level 1 
Refinement level 2 
Refinement level 2 


Patch 0 


Patch 1 ( 


Fragment 0 
Fragment 1 


Patch 2 Fragment 2 


Scalar-, vector-, Tensorfield 


Fig. 4 Hierarchical structure of the data layout of the concept of a field in computer memory: (1) 
organization by multiple resolutions for same spatial domain; (2) multiple coordinate systems coy- 
ering different spatial domains (arbitrary overlap possible); (3) fragmentation of fields into blocks 
(recombination from parallel data sources); and (4) layout of compound fields as components for 
performance reasons, indicated as S (scalar field), {x, y, z} for vector fields, and {xx, xy, yy, yz, 
zz, zx} for tensor fields 


level. This will be explained in more detail in section “Topological Skeletons.” 
The scheme also extends to cases beyond the purely mathematical basis to also 
cover data sets that occur in praxis, which is described in section “Non-topological 
representations.” A representation is identified via some reference object, which 
may be some coordinate system or another Skeleton. The lowest levels of fragments 
and compounds describe the internal memory layout of a Field data set and are 
optional; some examples are described in [8, 9]. 


Field Properties 

A specific Field identifier may occur in multiple locations. All these locations 
together define the properties of a field. The following four properties are express- 
ible in the data model: 


1. Hierarchical ordering: For a certain point in space, there exist multiple data 
values, one for each refinement level. This property describes the topological 
structure of the base space. 

2. Multiple coordinate systems: One spatial point may have multiple data repre- 
sentations relating to different coordinate systems. This property describes the 
geometrical structure of the base space. 

3. Fragmentation: Data may stem from multiple sources, such as a distributed 
multiprocess simulation. The field then consists of multiple data blocks, each 
of them covering a subdomain of the field’s base space. Such field fragments 
may also overlap, known as “ghost zones.” 

4. Separated Compounds: A compound data type, such as a vector or tensor, may be 
stored in different data layouts since applications have their own preferences. An 
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array of tensors may also be stored as a tensor of arrays, e.g., XYZXYZXYZXYZ 
as XXXXYYYYZZZZ. This property describes the internal structure of the fiber 
space. 


All of these properties are optional. In the most simple case, a field is just 
represented by an array of native data types; however, in the most general case 
(which the visualization algorithm must always support), the data are distributed 
over several such property elements and built from many arrays. With respect to 
quick transfer to the GPU, only the ability to handle multiple arrays per data set is 
of relevance. 

Figure 4 illustrates the organization of the last four levels of the data model. 
These consist of Skeleton and Representation objects with optional fragmentation 
and compound levels. The ordering of these levels is done merely based on their 
semantic importance, with the uppermost level (1) embracing multiple resolutions 
of the spatial domain being the most visible one to the end user. Each of these 
resolution levels may come with different topological properties, but all arrays 
within the same resolution are required to be topologically compatible (i.e., share 
the same number of points). There might still be multiple coordinate representations 
required for each resolution, which constitutes the second hierarchy level (2) of 
multiple coordinate patches. Data per patch may well be distributed over various 
fragments (3), which is considered an internal structure of each patch, due to 
parallelization or numerical issues, but not fundamental to the physical setup. Last 
but not least, fields of multiple components such as vector or tensor fields may be 
separated into distinct arrays themselves [7]. This property, merely a performance 
issue of in-memory data representation, is not what the end user usually does not 
want to be bothered with and is thus set as the lowest level in among these four 
entries. 


Topological Skeletons 

The Skeleton level of the fiber bundle hierarchy describes a certain topological 
property. This can be the vertices, the cells, the edges, etc. Its primary purpose 
is to describe the skeletons of a CW complex, but they may also be used 
to specify mesh refinement levels and agglomerations of certain elements. All 
data fields that are stored within a Skeleton level provide the same number of 
elements. In other words they share their index space (a data space in HDF5 
terminology). Each Topology object within a Grid object is uniquely identified 
via a set of integers, which are the dimension (e.g., the dimension of a k-cell), 
index depth (how many dereferences are required to access coordinate information 
in the underlying manifold), and refinement level (a multidimensional index, 
in general). Vertices — index depth 0 — of a topological space of dimension 
n define a Skeleton of type (n, 0). Edges are one-dimensional sets of vertex 
indices; therefore, their index depth is 1 and their Skeleton type is (1,1). Faces 
are two-dimensional sets of vertex indices, hence Skeleton type (2, 1). Cells — 
such as a tetrahedron or hexahedra — are described by a Skeleton type (3, 1). 


2118 W. Benger et al. 


MMs Atmospheric model 


Temperature 


oO 
a 
=) 
n 
no 
co) 
o 
ao 


ADCIRC storm-surge model 
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Fig. 5 The five-level organization scheme used for atmospheric data (MMS5 model data) and 
surge data (ADCIRC simulation model), built upon common topological property descriptions 
with additional fields (From Venkataraman et al. [81]) 


All the Skeleton objects of index depth 1 build the k-skeletons of a manifold’s 
triangulation. 

Higher index depths describe sets of k-cells. For instance, a set of edges describes 
a line — a path along vertices in a Grid. Such a collection of edges will fit into a 
Skeleton of dimension | and depth 2, i.e., type (1, 2). It is a one-dimensional object 
of indices that refer to edges that refer to vertices. 


Non-topological Representations 

Polynomial coordinates, information on field fragments, histograms, and color maps 
can be formulated in the fiber bundle model as well. These quantities are no longer 
direct correspondences of the mathematical background, but they may still be cast 
into the given context. 

Coordinates may be given procedurally, such as via some polynomial expression. 
The data for such expressions may be stored in a Skeleton of negative index depth — 
as these data are required to compute the vertex coordinates and more fundamental 
than these in this case. 
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A fragment of a Field given on vertices — the (1, 0)-Skeleton of a Grid — defines an 
n-dimensional subset of the Grid, defined by the hull of the vertices corresponding to 
the fragments. These may be expressed as a (n, 2)-Skeleton, where the Field object 
named “Positions” (represented relative to the vertices) refers to the (global) vertex 
indices of the respective fragments. The representation in coordinates corresponds 
to its range, known as the bounding box. Similarly, a field given on the vertices will 
correspond to the field’s numerical minimum/maximum range within this fragment. 

A histogram is the representation of a field’s vertex complex in a “chart” 
describing the required discretization, depending on the min/max range and a 
number count. A color map (transfer function) can be interpreted as a chart object 
itself. It has no intrinsically geometrical meaning, but provides means to transform 
some data. For instance, some scalar value will be transformed to some RGB triple 
using some color map. A scalar field represented in a certain color map is therefore 
of type RGB values and could be stored as an array of RGB values for each vertex. 
In practice, this will not be done since such transformation is performed in real time 
by modern graphics hardware. However, this interpretation of a color map as a chart 
object tells how color maps may be stored in the fiber bundle data model. 


3 Differential Forms and Topology 


This section not only introduces the concepts of differential forms and their discrete 
counterparts but also illustrates that similar concepts are applied in several separate 
areas of scientific visualization. Since the available resources are discrete and finite, 
concepts mirroring these characteristics have to be applied to visualize complex data 
sets. The most distinguished algebraic structure is described by exterior algebra (or 
Grassmann algebra, see also section “Exterior Product’), which comes with two 
operations, the exterior product (or wedge product) and the exterior derivative. 


Differential Forms 


Manifolds can be seen as a precursor to model physical quantities of space. Charts 
on a manifold provide coordinates, which allows using concepts which are already 
well established. Furthermore, they are crucial for the field of visualization, as they 
are key components to obtain depictable expressions of abstract entities. Tangential 
vectors were already introduced in section “Tangential Vectors” as derivatives along 
a curve. Then a one-form a is defined as a linear mapping which assigns a value 
to each tangential vector v from the tangent space Tp(M),ie.,a@ : Tp(M) > R. 
They are commonly called co-variant vectors, covectors (see section “Tangential 
Vectors”), or Pfaff-forms. The set of one-forms generates the dual vector space 
or cotangential space ie (M). It is important to highlight that the tangent vectors 
v € Tp(M) are not contained in the manifold itself, so the differential forms also 
generate an additional space over P € M. In the following, these one-forms are 
generalized to (alternating) differential forms. 
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An alternative point of view treats a tangential vector v as a linear mapping 
which assigns a scalar to each one-form a by < a,v >e€ R. By omitting one of 
the arguments of the obtained mappings, < a,. > or a(v) and <., v > or v(q@), 
linear objects are defined. Multi-linear mappings depending on multiple vectors or 
covectors appear as an extension of this concept and are commonly called tensors 


yiT'™<T" +R (26) 


where n and m are natural numbers and 7” and T*” represent the n and m powered 
Cartesian product of the tangential space or the dual vector space (cotangential 
space). A tensor y is called an (1, m)-tensor which assigns a scalar value to a set of 
m covectors and n vectors. All tensors of a fixed type (n, m) generate a tensor space 
attached at the point P € M. The union of all tensor spaces at the points P ¢ M 
is called a tensor bundle. The tangential and cotangential bundles are specialized 
cases for (1, 0) and (0, 1) tensor bundles, respectively. Fully antisymmetric tensors 
of type (0, m) may be identified with differential forms of degree m. For m > 
dim(M), where dim(/) represents the dimension of the manifold, differential 
forms vanish. 

The exterior derivative or Cartan derivative of differential forms generates a p + 
1-form df from a p-form f and conforms to the following requirements: 


1. Compatibility with the wedge product (product rule): d(a A B) = da A B + 
(-1)"a A dB 

2. Nilpotency of the operation d, d od = 0, depicted in Fig. 11 

3. Linearity 

A subset of one-forms is obtained as a differential df of zero-forms (functions) 

f at P and are called exact differential forms. For an n-dimensional manifold M, 
a one-form can be depicted by drawing (n — 1)-dimensional surfaces, e.g., for the 
three-dimensional space, Fig. 6 depicts a possible graphical representation of a 
one-form attached to M. This depiction also enables a graphical representation 
on how to integrate differential forms, where only the number of surfaces which 
are intersected by the integration domain has to be counted: 


<dfiv >= df (v) =a(v) (27) 


A consequence of being exact includes the closeness property da = 0. 
Furthermore, the integral Cy df with C, representing an integration domain, e.g., 
an interval x; and x2, results in the same value f(x2) — f(x,). In the general 
case, a p-form is not always the exterior derivative of a p-one-form; therefore, the 
integration of p-forms is not independent of the integration domain. An example is 
given by the exterior derivative of a p-form 6 resulting ina p + 1-form y = dB. 
The structure of such a generated differential form can be depicted by a tube-like 
structure such as in Fig.7. While the wedge product of an r-form and an s-form 
results in an r + s-form, this resulting form is not necessarily representable as a 
derivative. Figure 7 depicts a two-form which is not constructed by the exterior 
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Fig. 6 Possible graphical 
representation of the 
topological structure of 
one-forms in three 
dimensions. Note that the 
graphical display of 
differential forms varies in 
different dimension and does 
not depend on the selected 
basis elements 


Fig. 7 Possible graphical 
representation of a general 
two-form generated by a A B, 
where a and f are one-forms. 
The topologically tube-like 
structure of the two-forms is 
enclosed by the depicted 
planes 


derivative but instead by a A 6, where a and f are one-forms. In the general case, a 
p-form attached on an n-dimensional manifold M is represented by using (n — p)- 
dimensional surfaces. 

By sequentially applying the operation d to (0, m) for 0 < m < dim(M), 
the de Rham complex is obtained, which enables the investigation of the relation 
of closed and exact forms. The de Rham complex enables the transition from 
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the continuous differential forms to the discrete counterpart, so-called cochains. 
The already briefly mentioned topic of integration of differential forms is now 
mapped onto the integration of these cochains. To complete the description, the 
notion of chains, also modeled by multivectors (as used in Geometric Algebra, see 
section “Geometric Algebra” and Sect. 4) or fully antisymmetric (n, 0)-tensors, as 
description of integration domains is presented, where a chain is a collection of 
n-cells. 

The connection between chains and cochains is investigated in algebraic topol- 
ogy under the name of homology theory, where chains and cochains are collected in 
additive Abelian groups C, (M). 


Chains 

The de Rham complex collects cochains similar to a cell complex aggregating cells 
as elements of chains. To use these elements, e.g., all edges, in a computational 
manner, a mapping of the n-cells onto an algebraic structure is needed. An algebraic 
representation of the assembly of cells, an n-chain, over a cell complex K and a 
vector space Y can be written by 


j a 
Ch = y wit, T,€ K,w, eV 


i=l 
which is closed under reversal of the orientation: 
i : i 
Vt, €c, there is—t, € cy 


The different topological elements are called cells, and the dimensionality is 
expressed by adding the dimension such as a three-cell for a volume, a two-cell for 
surface elements, a one-cell for lines, and a zero-cell for vertices. If the coefficients 
are restricted to {—1,0, 1} € Z, the following classification for elements of a cell 
complex is obtained: 


¢ 0: if the cell is not in the complex 
e 1: if the unchanged cell is in the complex 
¢ —1:if the orientation is changed 


The so-called boundary operator is a map between sets of chains C, on a 
cell complex K. Let us denote the ith p-cell as tT, = ko,...kp, whereby 
t, € K. The boundary operator 0, defines a (p — 1)-chain computed 
from a p-chain: 0, : C,(K) --»> C,-1(K). The boundary of a cell 
t) can be written as alternating sum over elements of dimension p — 
1: 


in = » (1) lke, Rises, Rises Kel (28) 


Differential Methods for Multi-dimensional Visual Data Analysis 2123 


Cg 


C4 Co 


as 


A Cy 
B 


Fig. 8 Representation of a one-chain 1) with zero-chain boundary rd (left) and a two-chain 12 
with one-chain boundary cai (right) 


where k; indicates that k; is deleted from the sequence. This map is compatible with 
the additive and the external multiplicative structure of chains and builds a linear 
transformation: 


COs (29) 


Therefore, the boundary operator is linear 


8 (D>, with) = D> wi (9z}) (30) 


which means that the boundary operator can be applied separately to each cell 
of a chain. Using the boundary operator on a sequence of chains of different 
dimensions results in a chain complex C, = {C,,0,} such that the complex 


property 
0,210, = 0 (31) 


is given. Homological concepts are visible here for the first time, as homology 
examines the connectivity between two immediately neighboring dimensions. 
Figure 8 depicts two examples of one-chains and two-chains and an example of 
the boundary operator. 

Applying the appropriate boundary operator to the two-chain example reads 


dom =ti +17 + +7} (33) 
At+7+ptmM=At+n-wR+A-G+oi-A-—71d =0 
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Fig. 9 Examples of violations of correct cell attachment. Left: missing zero-cell. Middle: cells do 
not intersect at vertices. Right: intersection of cells 


A different view on chain complexes presents itself when the main focus is placed 
on the cells within a chain. To cover even the most abstract cases, a cell is defined 
as a subset c C X of a Hausdorff space X if it is homeomorphic to the interior of 
the open n-dimensional ball D” = {x € R” : |x| < 1}. The number n is unique 
due to the invariance of domain theorem [13] and is called the dimension of c, 
whereas homeomorphic means that two or more spaces share the same topological 
characteristics. The following list assigns terms corresponding to other areas of 
scientific computing: 


¢ 0-cell: point 
e 1-cell: edge 
e 2-cell: facet 
¢ n-cell: cell 


A cell complex K (see also section “Topology: Discretized Manifolds”) can be 
described by a set of cells that satisfy the following properties: 


* The boundary of each p-cell t, is a finite union of (p — 1)-cells in K : Op t= 
Ly mn : 
p-l : 
¢ The intersection of any two cells cs t in K is either empty or is a unique cell in 


K. 


The result of these operations are subspaces X) which are called the n-skeletons 
of the cell complex. Incidence and adjacence relations are then available. Examples 
for incidence can be given by vertex on edge relation and for adjacency by vertex to 
vertex relations. This cell complex with the underlying topological space guarantees 
that all interdimensional objects are connected in an appropriate manner. Although 
there are various possible attachments of cells, only one process results in a cell 
complex, see Fig. 9. 


Cochains 
In addition to chain and cell complices, scientific visualization requires the notation 
and access mechanisms to global quantities related to macroscopic n-dimensional 


Differential Methods for Multi-dimensional Visual Data Analysis 2125 


space-time domains. The differential forms which are necessary concepts to handle 
physical properties can also be projected onto discrete counterparts, which are 
called cochains. This collection of possible quantities, which can be measured, 
can then be called a section of a fiber bundle, which permits the modeling of 
these measurements as a function that can be integrated on arbitrary n-dimensional 
(sub)domains or multivectors. This function can then be seen as the abstracted 
process of measurement of this quantity [55,75]. The concept of cochains allows 
the association of numbers not only to single cells, as chains do, but also to 
assemblies of cells. Briefly, the necessary requirements are that this mapping is not 
only orientation dependent but also linear with respect to the assembly of cells. A 
cochain representation is now the global quantity association with subdomains of a 
cell complex, which can be arbitrarily built to discretize a domain. 

A linear transformation o of the n-chains into the field R of real numbers forms a 
vector space c, —> A{o}R and is called a vector-valued m-dimensional cochain or 


short m-cochain. The coboundary 6 of an m-cochain is an (m + 1)-cochain defined 
as 


bc" = pa Viti, Where vj = ee faces(z)) © 0 Tj )Cm(B) (34) 


Thus, the coboundary operator assigns nonzero coefficients only to those (m + 1) 
cells that have cy, as a face. As can be seen, 5c, depends not only on c, but on 
how c» lies in the complex K. This is a fundamental difference between the two 
operators 0 and 6. An example is given in Fig. 10 where the coboundary operator is 
used on a one-cell. The right part 06K of Fig. 10 is also depicted for the continuous 
differential forms in Fig. 7. The coboundary of an m-cochain is an (m + 1)-cochain 
which assigns to each (m+ 1) cell the sum of the values that the m-+ 1-cochain assign 
to the m-cells which form the boundary of the (m + 1) cell. Each quantity appears in 
the sum multiplied by the corresponding incidence number. Cochain complices [33, 
35] are similar to chain complices except that the arrows are reversed, so a cochain 
complex C* = {C’”, 6’"} is a sequence of modules C” and homomorphisms: 


gm semis cmt (35) 
such that 


gmtlgm —0 (36) 


3 
K!— +6K!— +80 6K! = 0. Proceeding from left to right, a one-cochain 
represented by a line segment, a two-cochain generated by the product of two one- 
forms, and a three-cochain depicted by volume objects are illustrated. 
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Fig. 10 Cochain complex with the corresponding coboundary operator 


Then, the following sequence with 6 o 6 = 0 is generated: 
8 3 8 3 8 
0-0" s 30] SS Se = SB (37) 


Cochains are the algebraic equivalent of alternating differential forms, while the 
coboundary process is the algebraic equivalent of the external derivative and can 
therefore be considered as the discrete counterpart of the differential operators: 


* grad. 
¢ curl. 
e div. 


It indeed satisfies the property 6 o 6 = 0 corresponding to 


* curlgrad. = 0 
¢ divcurl. = 0 


Duality Between Chains and Cochains 
Furthermore, a definition of the adjoint nature of 0,8 : C? + C?*! can be given: 


(c?, 0Cp41) = (8e?, Cp 41) (38) 


The concepts of chains and cochains coincide on finite complices [45]. Geometri- 
cally, however, C, and C? are distinct [12] despite an isomorphism. An element 
of C, is a formal sum of p-cells, where an element of C? is a linear function 
that maps elements of C, into a field. Chains are dimensionless multiplicities of 
aggregated cells, whereas those associated with cochains may be interpreted as 
physical quantities [65]. The extension of cochains from single cell weights to 
quantities associated with assemblies of cells is not trivial and makes cochains 
very different from chains, even on finite cell complices. Nevertheless, there is an 
important duality between p-chains and p-cochains. The first part of the de Rham 
(cohomology group) complex, depicted in Fig. 11 on the left, is the set of closed 
one-forms modulo the set of exact one-forms denoted by 


H! = Z'/B} (39) 
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Fig. 11 A graphical 
representation of closed and 
exact forms. The forms Z,, 


B,, Zo, and Bo are closed 
forms, while only the forms 
Bo and B, are exact forms. 
The nilpotency of the 
operation d forces the exact 
XK 


forms to vanish 


0 0 0 0 


This group is therefore trivial (only the zero element) if all closed one-forms are 
exact. If the corresponding space is multiply connected, then there are closed one- 
chains that are not themselves boundaries, and there are closed one-forms that are 
not themselves exact. 

For a chain cy, € C,(K,R) and a cochain c? € C?(K,R), the integral of c? 
over Cy is denoted by de c’, and integration can be regarded as a mapping, where 


D represents the corresponding dimension: 
[cro xcre) >R, forO< p<D (40) 


Integration in the context of cochains is a linear operation: given a,,a2 € R, 
cPleP? © C?(K) andc, € C,(K), reads 


/ ajc?! + anc? =a f cPl +a, | cP (41) 
c € ¢ 


PB P PB 


Reversing the orientation of a chain means that integrals over that chain acquire the 
opposite sign 


using the set of p-chains with vector space properties C,(K,R), e.g., linear 
combinations of p-chains with coefficients in the field R. For coefficients in R, 
the operation of integration can be regarded as a bilinear pairing between p-chains 
and p-cochains. Furthermore, for reasonable p-chains and p-cochains, this bilinear 
pairing for integration is nondegenerate, 


cP= -| cP (42) 


Cp P 


if : c?=0 VepeC,(K), thenc? =0 (43) 


Cp 
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and 


if / cP=0 Vc? €C(K), thenc, = 0 (44) 


Cp 


The integration domain can be described by, using Geometric Algebra notation, 
the exterior product applied to multivectors. An example is then given by the 
generalized Stokes theorem: 


fa-fs (45) 


Pp 


or 
<dficp > = < fi dcp > (46) 


The generalized Stokes theorem combines two important concepts, the integration 
domain and the form to be integrated. 


Homology and Cohomology 


The concepts of chains can also be used to characterize properties of spaces, 
the homology and cohomology, where it is only necessary to use C,(K, Z). The 
algebraic structure of chains is an important concept, e.g., to detect a p-dimensional 
hole that is not the boundary of a p + 1-chain, which is called a p-cycle. For short, 
a cycle is a chain whose boundary is 0)c, = 0, a closed chain. The introduced 
boundary operator can also be related to homological terms. A boundary is a chain 
b, for which there is a chain c, such that 0,cp = by. Since 00d = 0, By C Zn 
is obtained. The homology is then defined by H, = Z,/B,. The homology of a 
space is a sequence of vector spaces. The topological classification of homology is 
defined by 


By = im dp41 and 
Zp = ker 0p 
so that B, C Z, and 
Hy = Z,/By 


where 8, = {Rank} H,. Here {im} is the image and {ker} is the kernel of the 
mapping. For cohomology 


B? =imd?*! and 
Z? =ker d? 
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Fig. 12 Topologically a 
torus is the product of two 
circles. The partially shaded 
circle is spun around the fully 
drawn circle which can be 
interpreted as the closure of a 
cylinder onto itself 


so that B? C Z? and 


H? = Z?/B? 


where 8? = {Rank} H”. An important property of these vector spaces is given 
by f, which corresponds to the dimension of the vector spaces H and is called the 
Betti number [35, 86]. Betti numbers identify the number of nonhomologous cycles 
which are not boundaries: 


¢ Bo counts the number of connected components. 
¢ 6, counts the number of tunnels (topological holes). 
¢ > counts the number of enclosed cavities. 


The number of connected components gives the number of distinct entities of a 

given object, whereas tunnels describe the number of separated parts of space. In 

contrast to a tunnel, the enclosed cavities are completely bounded by the object. 
Examples for the Betti numbers of various geometrical objects are stated by: 


¢ Cylinder: Bp = 1,8; = 1,8, = OWn > 2. The cylinder consists of one 
connected component, which forms a single separation of space. Therefore no 
enclosed cavitiy is present. 

¢ Sphere: Bo = 1,6; = 0, 82 = 1, B, = OWn > 3. If 6; and B2 are switched, a 
sphere is obtained by contracting the separation by generating an enclosed cavity 
from the tunnel. 

¢ Torus: Bp = 1,6; = 2,62 = 1,8, = OWn > 3. Closing a cylinder onto 
itself results in a torus which not only generates an enclosed cavity but also 
maintains the cylinder’s tunnel. An additional tunnel is introduced due to the 
closing procedure which is depicted in Fig. 12 as the central hole. 


The Euler characteristics, which is an invariant, can be derived from the Betti 
numbers by: €& = Bo — Bi + Bo. 
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Fig. 13 A graphical representation of (co)homology for a three-dimensional cell complex 


0 0 0 0 


Fig. 14 Illustration of cycles 
A, B,C and a boundary C. 
A, B are not boundaries 


Figure 13 depicts the homology of a three-dimensional chain complex with the 
respective images and kernels, where the chain complex of K is defined by {im} 
Oy4+1 © {ker} 0,. As can be seen, the boundary operator expression yields 0, 0 


a 


pti = 0. 

To give an example, the first homology group is the set of closed one-chains 
(curves) modulo the closed one-chains which are also boundaries. This group is 
denoted by H; = Z,/B,, where Z, are cycles or closed one-chains and B, are 
one-boundaries. Another example is given in Fig. 14, where A, B, C are cycles and 
a boundary C, but A, B are not boundaries. 


Topology 


Conceptual consistency in scientific visualization is provided by topology. Cell 
complices convey topology in a computationally treatable manner and can therefore 
be introduced by much simpler definitions. A topological space (X,7) is the 
collection of sets 7 that include: 
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Fig. 15 Topologically a torus and a coffee mug are equivalent and so have the same Betti numbers 


¢ The space itself X and the empty set 0 
¢ The union of any of these sets 
¢ The finite intersection of any of the sets 


The family 7 is called a topology on X, and the members of 7 are called open 
sets. As an example a basic set X = {a,b,c} and a topology is given: 


(X,T)={ 9%, 
{a}, {D}, tc}, 
{a,b}, {a,c}, {B, c}, 
{a, b, c}} 


The general definition for a topological space is very abstract and allows several 
topological spaces which are not useful in scientific visualization, e.g., a topological 
space (X,7) with a trivial topology JT = {@,X}. So basic mechanisms of 
separation within a topological space are required, e.g., the Hausdorff property. A 
topological space (X,7) is said to be Hausdorff if, given x,y € X with x # y, 
there exist open sets U,, U2 such that x € U,, y € Uz and U; MN Uz = Q. But the 
question remains on what “topology” actually is. A brief explanation is given by the 
study of properties of an object that do not change under deformation. To describe 
this deformation process, abstract rules can be stated and if they are true, then an 
object A can be transformed into an object B without change. The two objects A, B 
are then called homeomorphic: 


¢ All points of A < all points of B 

¢ 1-—1 correspondence (no overlap) 
¢ Bicontinous (continuous both ways) 
¢ Cannot tear, join, poke/seal holes 
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The deformation is | — | if each point of A maps to a single point on B and 
there is no overlap. If this deformation is continuous, A cannot be teared, joined, 
disrupted, or sealed up. If two objects are homeomorphic, then they are topologically 
equivalent. Figure 15 illustrates an example of a torus and coffee mug which are 
a prominent example for topological equivalence. The torus can be continuously 
deformed, without tearing, joining, disrupting, or sealing up, into a cup. The 
hole in the torus becomes the handle of the cup. But why should anybody in 
visualization be concerned about how objects can be deformed? Topology is much 
more than the illustrated properties, it can be much better described by the study of 
connectedness: 


¢ Understanding of space properties: how connectivity happens. 

e Analysis of space properties: how connectivity can be determined. 

e Articulation of space properties: how connectivity can be described. 
¢ Control about space properties: how connectivity can be enforced. 


Topology studies properties of sets that do not change under well-behaved 
transformations (homeomorphisms). These properties include completeness and 
compactness. In visualization, one property is of significance: connectedness. 
Especially, how many disjoint components can be distinguished and how many 
holes (or tunnels) are in these components. Geometric configuration is another 
interesting aspect in visualization because it is important to know which of these 
components have how many holes, and where the holes are relative to each other. 
Several operations in scientific visualization can be summarized: 


¢ Simplification: reduction of data complexity. If objects are described with 
fewer properties, important properties such as components or holes should be 
retained or removed, if these properties become insignificant, unnecessary, or 
imperceptible. 

¢ Compression: reduction of data storage. It is important that each operation does 
not alter important features (interaction of geometrical and topological features). 

e Texturing: visualization context elements. How can a texture kept consistent if 
an object, e.g., a torus, is transformed into another object, e.g., a coffee cup. 

¢ Morphing: transforming one object into another. If an object is morphed into 
another, topological features have to remain, e.g., the torus hole has to become 
the coffee cup handle hole. 


4 Geometric Algebra Computing 


Geometric Algebra as a general mathematical system unites many mathematical 
concepts such as vector algebra, quaternions, Pliicker coordinates, and projective 
geometry, and it easily deals with geometric objects, operations, and transfor- 
mations. A lot of applications in computer graphics, computer vision, and other 
engineering areas can benefit from these properties. In a ray-tracing application, for 
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rayR 


“Inter section = Ry S 


bounding sphere S 


Fig. 16 Spheres and lines are basic entities of Geometric Algebra to compute with. Operations 
like the intersection of them are easily expressed with the help of their outer product. The result 
of the intersection of a ray and a (bounding) sphere is another geometric entity, the point pair of 
the two points of the line intersecting the sphere. The sign of the square of the point pair easily 
indicates whether there is a real intersection or not 


instance, the intersection of a ray and a bounding sphere is needed. According to 
Fig. 16, this can be easily expressed with the help of the outer product of these two 
geometric entities. 

Geometric Algebra is based on the work of Hermann Grassmann (see the 
conference [62] celebrating his 200th birthday in 2009) and William Clifford 
[20, 21]. Pioneering work has been done by David Hestenes, who first applied 
Geometric Algebra to problems in mechanics and physics [39, 40]. 

The first time Geometric Algebra was introduced to a wider computer graphics 
audience was through a couple of courses at the SIGGRAPH conferences in 
2000 and 2001 (see [57]) and later at the Eurographics [41]. Researchers at the 
University of Cambridge, UK, have applied Geometric Algebra to a number of 
graphics-related projects. Geomerics [71] is a start-up company in Cambridge 
specializing in simulation software for physics and lighting, which presented 
its new technology allowing real-time radiosity in videogames utilizing com- 
modity graphics-processing hardware. The technology is based on Geometric 
Algebra wavelet technology. Researchers at the University of Amsterdam, the 
Netherlands, are applying their fundamental research on Geometric Algebra to 
3D computer vision and to ray tracing and on the efficient software imple- 
mentation of Geometric Algebra. Researchers from Guadalajara, Mexico, are 
primarily dealing with the application of Geometric Algebra in the field of 
computer vision, robot vision, and kinematics. They are using Geometric Algebra, 
for instance, for tasks like visual-guided grasping, camera self-localization, and 
reconstruction of shape and motion. Their methods for geometric neural com- 
puting are used for tasks like pattern recognition [5]. Registration, the task of 
finding correspondences between two point sets, is solved based on Geometric 
Algebra methods in [65]. Some of their kinematics algorithms are dealing with 
inverse kinematics, fixation, and grasping as well as with kinematics and differ- 
ential kinematics of binocular robot heads. At the University of Kiel, Germany, 
researchers are applying Geometric Algebra to robot vision and pose estimation 
[66]. They also do some interesting research dealing, for instance, with neu- 
ral networks based on Geometric Algebra [14]. In addition to these examples, 
there are many other applications like Geometric Algebra Fourier transforms for 
the visualization and analysis of vector fields [24] or classification and clus- 
tering of spatial patterns with Geometric Algebra [63] showing the wide area 
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Table 1 Multiplication table of the 2D Geometric Algebra. This algebra consists of basic 
algebraic objects of grade (dimension) 0, the scalar; of grade 1, the two basis vectors e; and e9; 
and of grade 2, the bi-vector e; A e2, which can be identified with the imaginary number 7 squaring 
to—l 


1 e| eo ej Ae 
1 1 ey eo €; Ae 
e| e| 1 €;) Aes eo 
eo eo —e; Ae, 1 —e| 
ei Ae ej Aeo —e2 el —1 


Table 2 List of the basic geometric primitives provided by the 5D conformal Geometric Algebra. 
The bold characters represent 3D entities (x is a 3D point, n is a 3D normal vector, and x” is 
the scalar product of the 3D vector x). The two additional basis vectors e9 and @o9 represent the 
origin and infinity. Based on the outer product, circles and lines can be described as intersections 
of two spheres, respectively two planes. The parameter r represents the radius of the sphere and 
the parameter d the distance of the plane to the origin 


Entity Representation 

Point P=xt+ $x C00 + e@ 
Sphere S=P— ST eo0 
Plane m=nt+ deco 

Circle Z=S,A $2 

Line L=2Am 


of possibilities of advantageously using this mathematical system in engineering 
applications. 


Benefits of Geometric Algebra 


As follows, we highlight some of the properties of Geometric Algebra that make it 
advantageous for a lot of engineering applications. 


Unification of Mathematical Systems 
In the wide range of engineering applications, many different mathematical systems 
are currently used. One notable advantage of Geometric Algebra is that it subsumes 
mathematical systems like vector algebra, complex analysis, quaternions, or Pliicker 
coordinates. Table 1, for instance, describes how complex numbers can be identified 
within the 2D Geometric Algebra. This algebra does not only contain the two basis 
vectors €; and e» but also basis elements of grade (dimension) 0 and 2 representing 
the scalar and imaginary part of complex numbers. 

Other examples are Pliicker coordinates based on the description of lines in 
conformal geometric algebra (see section “Conformal Geometric Algebra’) or 
quaternions as to be identified in Fig. 19 with their imaginary units. 
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Fig. 17 Spheres and circles 
are basic entities of 


Geometric Algebra. Sphere S1 
Operations like the : 
intersection of two spheres Circle = Sj» So 
are easily expressed 

Sphere So 


Uniform Handling of Different Geometric Primitives 

Conformal Geometric Algebra, the Geometric Algebra of conformal space we 
focus on, is able to easily treat different geometric objects. Table 2 presents the 
representation of points, lines, circles, spheres, and planes as the same entities 
algebraically. Consider the spheres of Fig. 17, for instance. These spheres are simply 
represented by 


1 
S = P51 esc (47) 


based on their center point P, their radius r, and the basis vector eg, which 
represents the point at infinity. The circle of intersection of the spheres is then easily 
computed using the outer product to operate on the spheres as simply as if they were 
vectors: 


Z=S,A S82 (48) 


This way of computing with Geometric Algebra clearly benefits computer 
graphics applications. 


Simplified Geometric Operations 
Geometric operations like rotations, translations (see [41]), and reflections can 
be easily treated within the algebra. There is no need to change the way of 
describing them with other approaches (vector algebra, for instance, additionally 
needs matrices in order to describe transformations). 

Figure 18 visualizes the reflection of the ray R from one plane 


m=n+ deg (49) 


(see Table 2). The reflected line, drawn in magenta, 


mR 
Reefiected == (50) 
IU 
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Fig. 18 The ray R is mR 
reflected from the plane z ,— — 
computing — =R 


T 


is computed with the help of the reflection operation including the reflection object 
as well as the object to be reflected. 


More Efficient Implementations 

Geometric Algebra as a mathematical language suggests a clearer structure and 
greater elegance in understanding methods and formulae. But, what about the 
runtime performance for derived algorithms? Geometric Algebra inherently has a 
large potential for creating optimizations leading to more highly efficient implemen- 
tations especially for parallel platforms. Gaalop [44], as presented in section “Com- 
putational Efficiency of Geometric Algebra Using Gaalop,” is an approach offering 
dramatically improved optimizations. 


Conformal Geometric Algebra 


Conformal Geometric Algebra is a 5D Geometric Algebra based on the 3D basis 
vectors €;, @2, and e3 as well as on the two additional base vectors eo representing 
the origin and é,o representing infinity. 

Blades are the basic computational elements and the basic geometric entities of 
geometric algebras. The 5D conformal Geometric Algebra consists of blades with 
grades (dimension) 0, 1, 2, 3, 4, and 5, whereby a scalar is a 0-blade (blade of 
grade 0). The element of grade five is called the pseudoscalar. A linear combination 
of blades is called a k-vector. So a bi-vector is a linear combination of blades with 
grade 2. Other k-vectors are vectors (grade 1), tri-vectors (grade 3), and quadvectors 
(grade 4). Furthermore, a linear combination of blades of different grades is called a 
multivector. Multivectors are the general elements of a Geometric Algebra. Table 3 
lists all the 32 blades of conformal Geometric Algebra. The indices indicate 1, 
scalar; 2-6, vector; 7-16, bi-vector; 17—26, tri-vector; 27-31, quadvector; and 32, 
pseudoscalar. 

A point P = x,e); + x2e2 +.x3e3 + 5X00 + eo (see Table 2), for instance, can be 
written in terms of a multivector as the following linear combination of blades b[7]: 
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Table 3 The 32 blades of Index Blade Grade 

the 5D conformal Geometric 

eae es 
2 l 
3 e2 1 
——— 
5 Co0 1 
6 1 
7 €} Neo 2 
— 
9 Jeep +i 
10 [eae *&2 
11 2 
12 2 
13 2 
14 2 
15 2 
16 2 
7 3 
18 3 
19 €; AN €2 A €9 3 
20 €) A e3 A Coo 3 
21 €; AN e3 A eg 3 
22 €) A loo A €0 3 
23 2 A @3 A €go 3 
24 €o A e3 Ae 3 
25 deAemwAe (3 
26 Jeshemae (3 
27 €) AN 2 N€3 A Coo 4 
28 €ji Nn Nez A & 4 
29 C1 N02 Ngo A&Q 4 
30 4 
31 4 
3 der NesAes Aeow Ae 

1 2 


with multivector indices according to Table 3. 

Figure 19 describes some interpretations of the 32 basis blades of conformal 
Geometric Algebra. Scalars like the number z are grade 0 entities. They can be 
combined with the blade representing the imaginary unit i to complex numbers 
or with the blades representing the imaginary units 7, 7, k to quaternions. Since 
quaternions describe rotations, this kind of transformation can be handled within the 
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Grade|Term Blades nr. 


cos a -sin a 0" 
sing cosa 0 
0 0 1 


4 Quadvector 6, A @, A &A ey, 5 
@, A 0, A &A Oy, 
nA OA OHA Oy, 
5 Pseudoscalar) % “ © “ 04 4 9, 1 


J 


Fig. 19 The blades of conformal Geometric Algebra. Spheres and planes, for instance, are vectors. 
Lines and circles can be represented as bi-vectors. Other mathematical systems like complex 
numbers or quaternions can be identified based on their imaginary units i, j,k. This is why also 
transformations like rotations can be handled within the algebra 


Table 4 The extended list of the two representations of the conformal geometric entities. The 
IPNS representations as described in Table | have also an OPNS representation, which are dual to 
each other (indicated by the star symbol). In the OPNS representation, the geometric objects are 
described with the help of the outer product of conformal points that are part of the objects, for 
instance, lines as the outer product of two points and the point at infinity 


Entity IPNS representation OPNS representation 
Sphere S* = P} A Po A P3 A P4 
Plane mw* = Pi A Po A P3 A Coo 
Circle Z* = Pi A Po A P3 

Line L* = P| A Po A €oo 
Point pair Pp* = P, A P 


algebra. Geometric objects like spheres, planes, circles, and lines can be represented 
as vectors and bi-vectors. 

Table 4 lists the two representations of the conformal geometric entities. The 
inner product null space (IPNS) and the outer product null space (OPNS) [61] 
are dual to each other. While Table 2 already presented the IPNS representation 
of spheres and planes, they can be described also with the outer product of four 
points being part of them. In the case of a plane one of these four points is 
the point at infinity e,5. Circles can be described with the help of the outer 
product of three conformal points lying on the circle or as the intersection of two 
spheres. 

Lines can be described with the help of the outer product of two points and 
the point at infinity e4, or with the help of the outer product of two planes (i.e., 
intersection in IPNS representation). An alternative expression is 
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Fig. 20 The line L through 
the 3D points a, b and the 
visualization of its 6D 
Pliicker parameters based on 
the two 3D vectors u and m 


of Eq. (53) 


L = e123 + MA 0 (52) 


with the 3D pseudoscalar e123 = e1 A €2 A @oo, the two 3D points a, b on the line, 
u = b—aas 3D direction vector, and m = a x b as the 3D moment vector (relative 
to origin). The corresponding six Pliicker coordinates (components of u and m) are 
(see Fig. 20) 


(u:m) = (wy : uy! ug: mM, 2 M2: m3) (53) 


Computational Efficiency of Geometric Algebra Using Gaalop 


Because of its generality, Geometric Algebra needs some optimizations for efficient 
implementations. 

Gaigen [27] is a Geometric Algebra code generator developed at the University of 
Amsterdam (see [23, 26]). The philosophy behind Gaigen 2 is based on two ideas: 
generative programming and specializing for the structure of Geometric Algebra. 
Please find some benchmarks comparing Gaigen 2 with other pure software 
solutions as well as comparing five models of 3D Euclidean geometry for a ray- 
tracing application in [26, 28]. 

Gaalop [44] combines the advantages of software optimizations and the adapt- 
ability on different parallel platforms. As an example, an inverse kinematics 
algorithm of a computer animation application was investigated [42]. With the 
optimization approach of Gaalop, the software implementation became three times 
faster and with a hardware implementation about 300 times faster [43] (three 
times by software optimization and 100 times by additional hardware optimization) 
than the conventional software implementation. Figure 21 shows an overview over 
the architecture of Gaalop. Its input is a Geometric Algebra algorithm written 
in CLUCalc [60], a system for the visual development of Geometric Algebra 
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Fig. 21 Architecture of Gaalop 


algorithms. Via symbolic simplification it is transformed into an intermediate 
representation (IR) that can be used for the generation of different output formats. 
Gaalop supports sequential platforms with the automatic generation of C and JAVA 
code while its main focus is on supporting parallel platforms like reconfigurable 
hardware as well as modern accelerating GPUs. 

Gaalop uses the symbolic computation functionality of Maple (using the Open 
Maple interface and a library for Geometric Algebras [1]) in order to optimize a 
Geometric Algebra algorithm. It computes the coefficients of the desired multivector 
symbolically, returning an efficient implementation depending just on the input 
variables. 

As an example, the following CLUCalc code computes the intersection circle C 
of two spheres S'1 and $2 according to Fig. 17: 


Pl = xlxel + x2*e2 + x3%e3 + 1/2% (X1*xX14+X%2%*xX24+xX3%xX3) xeinf 
+ e0; P2 = yleel + 

y2*e2 +y3*e3 + 1/2*(yleylt+y2«*y2+y3*y3) *«einf + e0; 
Sl =P1 - 1/2 * rlxrl « 

einf; S2 = P2 - 1/2 * r2*r2 * einf; ?C = S1 $\wedgesS S2; 


See Table 2 for the computation of the conformal points P 1 and P2, the spheres 
S1 and $2, as well as the resulting circle based on the outer product of the two 
spheres. 

The resulting C code generated by Gaalop for the intersection circle C is as 
follows and depends only on the variables x1, x2, x3, yl, y2, y3, rl, and r2 for 
the 3D center points and radii: 


float C [32] = {\{}o.0{\}}; CI7] = xl«*y2-x2«yl; 
C[8] = xlxy3-x3*yl; C[9] 

= -O0.5*yl*x1l*x1-0.5*yl*x2*x2 -0.5*yl*x3*x3+0.5*ylx*xrl*rl + 

O.5*xXleyl«*y1+0.5*xXl*ey2*y2 + 0.5*xX1lxy3x*y3 - 0.5*xX1*r2*r2; 
C[10] = -yl + 

x1; C[11] = -x3*y2+x2«*y3; C[12] = -0.5*y2«*x1l*x1-0.5*y2*x2«x2- 

O.5*y2*x3*x3 + O.5*y2*rl«erl + 0.5*x2*ylexyl + 0.5*x2*y2ey2 + 
0.5*x2*y3x*y3 - 

O.5*x2*r2*r2; C[13] = -y2 + x2; C[14] = -0.5*y3«*xl«*xl1 - 
0.5*y3*xX2*x2 
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-O.5*y3*xX3*x3 + O.5*y3*rlerl + 0.5*x3*xyleyl + O0.5%*x3*y2*y2 + 
0.5*x3*«y3x*y3 

- 0.5%x3*«r2*r2; C[15] = -y3 + x3; C[16] = -0.5xy3x*y3 + 
0.5*x3*x3 + 

O.54x24x2 + 0.5*r2*r2 -0.5*yl*eyl - O.5*y2*y2 + 0.5ex1lexl1 - 
O0.5x*rlxrl; 


In a nutshell, Gaalop always computes optimized 32-dimensional multivectors. 
Since a circle is described with the help of a bi-vector, only the blades 7-16 
(see Table 3) are used. As you can see, all the corresponding coefficients of this 
multivector are very simple expressions with basic arithmetic operations. 


5 Feature-Based Vector Field Visualization 


We will identify derived quantities that describe flow features such as vortices 
(section “Derived Measures of Vector Fields”) and we discuss the topology of 
vector fields (section “Topology of Vector Fields”). However, not all feature-based 
visualization approaches can be covered here. The reader is referred to [84] for 
further information on this topic. We start with a description of integral curves in 
vector fields, which are the basis for most feature-based visualization approaches. 


Characteristic Curves of Vector Fields 


Acurve g : R — M(see section “Tangential Vectors”’) is called a tangent curve of 
a vector field v(x), if for all points x € q the tangent vector g of g coincides with 
v(x). Tangent curves are the solutions of the autonomous ODE system 


£ x0) = v(x(t)) with x(0) = xo (54) 


For all points x € M with v(x) # 0, there is one and only one tangent curve 
through it. Tangent curves do not intersect or join each other. Hence, tangent curves 
uniquely describe the directional information and are therefore an important tool for 
visualizing vector fields. 

The tangent curves of a parameter-independent vector field v(x) are called 
streamlines. A streamline describes the path of a massless particle in v. 

In a one-parameter-dependent vector field v(x, t), there are four types of 
characteristic curves: streamlines, path lines, streak lines, and time lines. To ease the 
explanation, we consider v(x, ¢) as a time-dependent vector field in the following: 
in a space-time point (Xo, fo) we can start a streamline (staying in time slice t = f) 
by integrating 


“x(c) = v(x(T),f0) with x(0) = xo (55) 
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or a path line by integrating 


“x(0) = v(x(t),t) with x(t) = xo (56) 


Path lines describe the trajectories of massless particles in time-dependent vector 
fields. The ODE system (56) can be rewritten as an autonomous system at the 
expense of an increase in dimension by one, if time is included as an explicit state 


variable: 
d (x\ _ (v(x(t),1) . x _ (x0 
a(r)= (SP?) wm (To=(2) on 


In this formulation space and time are dealt with on equal footing — facilitating 
the analysis of spatiotemporal features. Path lines of the original vector field v in 
ordinary space now appear as tangent curves of the vector field 


p(x, 1) = eae (58) 


in space-time. To treat streamlines of v, one may simply use 


s(x, t) = Cc. : ’) (59) 


Figure 22 illustrates s and p for a simple example vector field v. It is obtained by 
a linear interpolation over time of two bilinear vector fields. Figure 22a depicts 
streamlines, Fig. 22b depicts pathlines. 

A streak line is the connection of all particles set out at different times but 
the same point location. In an experiment, one can observe these structures by 
constantly releasing dye into the flow from a fixed position. The resulting streak 
line consists of all particles which have been at this fixed position sometime in the 
past. Considering the vector field p introduced above, streak lines can be obtained in 
the following way: apply a stream surface integration in p where the seeding curve 
is a straight line segment parallel to the t-axis; a streak line is the intersection of this 
stream surface with a hyperplane perpendicular to the t-axis (Fig. 22c). 

A time line is the connection of all particles set out at the same time but different 
locations, 1.e., a line which gets advected by the flow. An analogon in the real world 
is a yarn or wire thrown into a river, which gets transported and deformed by the 
flow. However, in contrast to the yarn, a time line can get shorter and longer. It can 
be obtained by applying a stream surface integration in p starting at a line with t = 
{const.} and intersecting it with a hyperplane perpendicular to the t-axis (Fig. 22d). 

Streak lines and time lines cannot be described as tangent curves in the 
spatiotemporal domain. Both types of lines fail to have a property of stream and 
path lines: they are not locally unique, i.e., for a particular location and time there 
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v(x y, )=(1-t) 


Fig. 22 Characteristic curves of a simple 2D time-dependent vector field. Streamlines and path 
lines are shown as illuminated field lines. Streak and time lines are shown as thick cylindrical lines, 
while their seeding curves and resulting stream surfaces are colored red. The red/green coordinate 
axes denote the (x, y)-domain; the blue axis shows time 


is more than one streak and time line passing through. However, stream, path, and 
streak lines coincide for steady vector fields v(x, t) = v(x, fo) and are described 
by (54) in this setting. Time lines do not fit into this. 


Derived Measures of Vector Fields 


A number of measures can be derived from a vector field v and its derivatives. These 
measures indicate certain properties or features and can be helpful when visualizing 
flows. The following text assumes the underlying manifold M where the vector field 
is given to be Euclidean space, i.e., the manifold is three-dimensional and Cartesian 
coordinates are used where the metric (see section “Tensors’’) is representable as the 
unit matrix. 

The magnitude of v is then given as 


lv} = Vw +2 +0? (60) 


The divergence of a flow field is given as 
div(v) = V-v = trace(J) = uy + vy + w, (61) 


and denotes the gain or loss of mass density at a certain point of the vector field: 
given a volume element in a flow, a certain amount of mass is entering and exiting 
it. Divergence is the net flux of this at the limit of a point. A flow field with {div }(v) 
= 0 is called divergence-free, which is a common case in fluid dynamics since a 
number of fluids are incompressible. 
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The vorticity or curl of a flow field is given as 


@, Wy — Vz 
O=|)a }]=Vxv=]{ u-—wy (62) 
3 Vy — Uy 


This vector is the axis of locally strongest rotation, i.e., it is perpendicular to the 
plane in which the locally highest amount of circulation takes place. The vorticity 
magnitude |@| gives the strength of rotation and is often used to identify regions of 
high vortical activity. A vector field with w = 0 is called irrotational or curl-free, 
with the important subclass of conservative vector fields, i.e., vector fields which are 
the gradient of a scalar field. Note that Geometric Algebra (see section “Geometric 
Algebra” and Sect. 4) treats Eqs. (61) and (62) as an entity, called the geometric 
derivative. The identification of vortices is a major subject in fluid dynamics. The 
most widely used quantities for detecting vortices are based on a decomposition of 
the Jacobian matrix J = S + 92 into its symmetric part, the strain tensor 


S= sats") (63) 


and its antisymmetric part, the vorticity tensor 


1 0 —W3 W2 
Q= 5J-J') =| oa O -a (64) 
—a2 0, O 


with w; being the components of vorticity (62). While S2 assesses vortical activity, 
the strain tensor S measures the amount of stretching and folding which drives 
mixing to occur. 

Inherent to the decomposition of the flow field gradient J into S and £2 is the 
following duality: vortical activity is high in regions where S2 dominates S, whereas 
strain is characterized by S dominating £2. 

In order to identify vortical activity, Jeong et al. used this decomposition in 
[47] to derive the vortex region quantity A as the second largest eigenvalue of 
the symmetric tensor S? + 927. Vortex regions are identified by A. < 0, whereas 
A» > 0 lacks physical interpretation. 2 does not capture stretching and folding of 
fluid particles and hence does not capture the vorticity-strain duality. 

The Q-criterion of Hunt [46], also known as the Okubo-Weiss criterion, is 
defined by 


1 1 
Q==( 2 |? -S$ |?) =lo I? | SIP (65) 

2 2 
where Q is positive and the vorticity magnitude dominates the rate of strain. Hence 
it is natural to define vortex regions as regions where Q > 0. Unlike Ax, QO has a 
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physical meaning also where Q < 0. Here the rate of strain dominates the vorticity 
magnitude. 


Topology of Vector Fields 


In this section we collect the first-order topological properties of steady 2D and 3D 
vector fields. The extraction of these topological structures has become a standard 
tool in visualization for the feature-based analysis of vector fields. 


Critical Points 
Considering a steady vector field v(x), an isolated critical point xo is given by 


v(xo) = 90 with v(xo+ €) 40 (66) 


This means that v is zero at the critical point but nonzero in a certain neighborhood. 

Every critical point can be assigned an index. For a 2D vector field it denotes 
the number of counterclockwise revolutions of the vectors of v while traveling 
counterclockwise on a closed curve around the critical point (for 2D vector fields, 
it is therefore often called the winding number). Similarly, the index of a 3D 
critical point measures the number of times the vectors of v cover the area of 
an enclosing sphere. The index is always an integer and it may be positive or 
negative. For a curve/sphere enclosing an arbitrary part of a vector field, the 
index of the enclosed area/volume is the sum of the indices of the enclosed 
critical points. Mann et al. show in [54] how to compute the index of a region 
using Geometric Algebra. A detailed discussion of index theory can be found in 
(25,31, 32]. 

Critical points are characterized and classified by the behavior of the tangent 
curves around it. Here we concentrate on first-order critical points, i.e., critical 
points with det(J(xo)) # 0. As shown in [37, 38], a first-order Taylor expansion 
of the flow around xo suffices to completely classify it. This is done by an 
eigenvalue/eigenvector analysis of J(xo). Let A; be the eigenvalues of J(xo) ordered 
according to their real parts, ie., Re(A;-1) < Re(A;). Furthermore, let e; be the 
corresponding eigenvectors, and let f; be the corresponding eigenvectors of the 
transposed Jacobian (J(xo))” (note that J and J’ have the same eigenvalues but 
not necessarily the same eigenvectors). The sign of the real part of an eigenvalue 
4; denotes — together with the corresponding eigenvector e; — the flow direction: 
positive values represent an outflow and negative values an inflow behavior. Based 
on this we give the classification of 2D and 3D first-order critical points in the 
following. 


2D Vector Fields Based on the flow direction, first-order critical points in 2D 
vector fields are classified into: 
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Fig. 23 Classification of first-order critical points. R;, R2 denote the real parts of the eigenvalues 
of the Jacobian matrix, while /;, /. denote their imaginary parts (From [37]) 


Sources : 0 < Re(A1) < Re(A2) 
Saddles : Re(A,) <0 < Re(A2) 
Sinks : Re(A;) < Re(A2) < 0 


Thus, sources and sinks consist of complete outflow/inflow, while saddles have a 
mixture of both. 

Sources and sinks can be further divided into two stable subclasses by deciding 
whether or not imaginary parts are present, i.e., whether or not A,, A2 is a pair of 
conjugate complex eigenvalues: 


Foci: Im(A,) = —Im(A2) 4 0 
Nodes : Im(A,) = Im(A2) = 0 


There is another important class of critical points in 2D: a center. Here, we have 
a pair of conjugate complex eigenvalues with Re(A;) = Re(A2) = 0. This type is 
common in incompressible (divergence-free) flows but unstable in general vector 
fields since a small perturbation of v changes the center to either a sink or a source. 
Figure 23 shows the phase portraits of the different types of first-order critical points 
following [37]. 

The index of a saddle point is —1, while the index of a source, sink, or center 
is +1. It turns out that this coincides with the sign of det(J(xo)): a negative 
determinant denotes a saddle, a positive determinant a source, sink, or center. This 
already shows that the index of a critical point cannot be used to distinguish or 
classify them completely, since different types like sources and sinks have assigned 
the same index. 

An iconic representation is an appropriate visualization for critical points, since 
vector fields usually contain a finite number of them. We will display them as 
spheres colored according to their classification: sources will be colored in red, sinks 
in blue, saddles in yellow, and centers in green. 


3D Vector Fields Depending on the sign of Re(A;) we get the following classifica- 
tion of first-order critical points in 3D vector fields: 
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Fig. 24 Flow behavior around critical points of 3D vector fields and corresponding iconic 
representation 


Sources : 0 < 
Repelling saddles : Re(A;) < 0 < Re(A2) < Re(A3) 
Attracting saddles: Re(Ai) < Re(A2) <0 < Re(A3) 

Sinks : Re(A,) < Re(A2) < 


Again, sources and sinks consist of complete outflow/inflow, while saddles have 
a mixture of both. A repelling saddle has one direction of inflow behavior (called 
inflow direction) and a plane in which a 2D outflow behavior occurs (called outflow 
plane). Similar to this, an attracting saddle consists of an outflow direction and an 
inflow plane. 

Each of the four classes above can be further divided into two stable subclasses 
by deciding whether or not imaginary parts in two of the eigenvalues are present 
(A, Ao, A3 are not ordered): 


Foci : Im(A;)=0 and Im(A2) = —Im(A3) 4 0 
Nodes : Im(A,) = Im(A2) = Im(A3) = 0 


As argued in [29], the index of a first-order critical point is given as the sign of 
the product of the eigenvalues of J(xo). This yields an index of +1 for sources and 
attracting saddles and an index of —1 for sinks and repelling saddles. 

In order to depict 3D critical points, several icons have been proposed in the 
literature; see [30, 36, 37,53]. Basically, we follow the design approach of [72, 85] 
and color the icons depending on the flow behavior: attracting parts (inflow) are 
colored blue, while repelling parts (outflow) are colored red (Fig. 24). 


Separatrices 

Separatrices are streamlines or stream surfaces which separate regions of different 
flow behavior. Here we concentrate on separatrices that emanate from critical points. 
Due to the homogeneous flow behavior around sources and sinks (either a complete 
outflow or inflow), they do not contribute to separatrices. Each saddle point creates 
two separatrices: one in forward and one in backward integration into the directions 
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Fig. 25 Separatrices are streamlines or surfaces starting from saddle points into the direction of 
the eigenvectors 


of the eigenvectors. For a 2D saddle point, this gives two separation lines (Fig. 25a). 
Considering a repelling saddle xz of a 3D vector field, it creates one separation 
curve (which is a streamline starting in Xr in the inflow direction by backward 
integration) and a separation surface (which is a stream surface starting in the 
outflow plane by forward integration). Figure 25b gives an illustration. A similar 
statement holds for attracting saddles. 

Other kinds of separatrices are possible as well: they can emanate from boundary 
switch curves [85] and attachment and detachment lines [48], or they are closed 
separatrices without a specific emanating structure [73]. 


Application 

In the following, we exemplify the topological concepts described above by 
applying them to a 3D vector field. First, we extract the critical points by searching 
for zeros in the vector field. Based on an eigenvalue/eigenvector analysis, we 
identify the different types of the critical points. Starting from the saddles, we 
integrate the separatrices into the directions of the eigenvectors. 

Figure 26 visualizes the electrostatic field around a benzene molecule. This 
data set was calculated on a 101° regular grid using the fractional charges method 
described in [70]. It consists of 184 first-order critical points depicted in Fig. 26a. 
The separation surfaces shown in Fig.26b emanate from 78 attracting and 43 
repelling saddles. Note how they hide each other as well as the critical points. Even 
rendering the surfaces in a semitransparent style does not reduce the visual clutter 
to an acceptable degree. This is one of the major challenges for the topological 
visualization of 3D vector fields. 

Figure 26c shows a possible solution to this problem by showing the 129 saddle 
connectors that we found in this data set. Saddle connectors are the intersection 
curves of repelling and attracting separation surfaces and have been introduced 
to the visualization community in [72]. Despite the fact that saddle connectors 
can only indicate the approximate run of the separation surfaces, the resulting 
visualization gives more insight into the symmetry and three-dimensionality of the 
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Fig. 26 Topological representations of the benzene data set with 184 critical points. (a) Iconic 
representation. (b) Due to the shown separation surfaces, the topological skeleton of the vector 
field looks visually cluttered. (c) Visualization of the topological skeleton using saddle connectors 


data set. Saddle connectors are a useful compromise between the amount of coded 
information and the expressiveness of the visualization for complex topological 
skeletons. 


6 Anisotropic Diffusion PDEs for Image Regularization and 
Visualization 


Regularization PDEs: A Review 


We consider a 2D multivalued image I: {2 — R”(n = 3 for color images) defined 
ona domain Q C R? and denote by J; : 2 — R the scalar channel i of I: 


VX = (x,y) € Qey = ie Law --- Inco)’. 


Local Multivalued Geometry and Diffusion Tensors 

PDE-based regularization can be often seen as the local smoothing of an image 
I along defined directions depending themselves on the local configuration of the 
pixel intensities, i.c., one wants basically to smooth I in parallel to the image 
discontinuities. Naturally, this means that one has first to retrieve the Jocal geometry 
of the image I. It consists in the definition of these important features at each image 
point X = (x,y) € Q: 


¢ Two orthogonal directions Ox)t, Oxy € Ss! along the local maximum and 
minimum variations of image intensities at X. @~ is then considered to be parallel 
to the local edge, when there is one. 

¢ Two corresponding positive values Rew? A(x) measuring the effective varia- 
tions of the image intensities along Ax) * and 6x), respectively. A~, A* are 
related to the local contrast of an edge. 


For scalar images J : Q — R, this local geometry {A*/~,6*/-|K € Q} is 
usually retrieved by the computation of the smoothed gradient field V J, = 
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VI * Gz where G, is a 2D Gaussian kernel with standard deviation o. Then, 
A+ =|| VJ, ||? is a possible measure of the local contrast of the contours, 
while 0~ = VJ,+/||VJ,|| gives the contours direction. Such a local geometry 
{At/~,6+/-|X © Q} can be represented in a more convenient form by a field 
G: 2 — P(2) of second-order tensors (2 x 2 symmetric and semi-positive 
matrices): 


VXEQ, Gy =A" HAT FET, 


Eigenvalues of G are indeed A~ and At and corresponding eigenvectors are 67 
and @*. The local geometry of scalar-valued images J can be then modeled by the 
tensor field Gq) = View VI. Lee 

For multivalued images I: Q — R", the local geometry can be retrieved in a 
similar way, by the computation of the field G of the smoothed structure tensors. As 


explained in [22, 82], this is a nice extension of the gradient for multivalued images: 


ali 
VXEQ, Gow = (So, VliecoViiao), Go where Via = & *Ga 


(67) 
G, x) is a very good estimator of the local multivalued geometry of I at X: its spectral 
elements give at the same time the vector-valued variations (by the eigenvalues 
A-,At of G,) and the orientations (edges) of the local image structures (by the 
eigenvectors 0~ 16+ of G,),o being proportional to the so-called noise scale. 
Once the local geometry G, of I has been determined, the way the regularization 
process is achieved is defined by another field T: 2 — P(2) of diffusion tensors, 
which specifies the local smoothing geometry that should drive the PDE flow. Of 
course, T depends on the targeted application, and most of the time it is constructed 
from the local geometry G, of I. It is thus defined from the spectral elements 
A~,At and 6~,6* of G,. In [19, 78], the following expression is proposed for 
image regularization: 


WXEQ, Te = far) OO + fay ater" (68) 


where 


7 1 a 1 
Kaya) (14+ At +A7-)P1 ame 3 (Q+At +a-)m 


with pi < po 


are the two functions which set the strengths of the desired smoothing along the 
respective directions 9~, 6+. This latest choice basically says that if a pixel X is 
located on an image contour A® is high), the smoothing on X would be performed 


mostly along the contour direction Oe) (since fe «K fi). Conversely, if a pixel 


+ 


X is located on a homogeneous region Ow 


is low), the smoothing on X would be 
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performed in all possible directions (isotropic smoothing), since Fak o~ ti.) (and 
then T ~ I). Predefining the smoothing geometry T of each applied PDE iteration 
is the first stage of most of the PDE-based regularization algorithms. Most of the 
differences between existing regularization methods (as in [2,3, 10, 18, 19,49,52,58, 
59, 67—69]) lie first on the definition of T, but also on the kind of the diffusion PDE 
that will be used indeed to perform the desired smoothing. 


Divergence-Based PDEs 
One of the common choices to smooth a corrupted multivalued image I:Q — R” 
following a local smoothing geometry T: {2 — P(2) is to use the divergence PDE: 


Tj 
a ee at = div(TVI,) (69) 


The general form of this now classical PDE for image regularization has been intro- 
duced by Weickert in [82] and adapted for color/multivalued images in [83]. In this 
latter case, the tensor field T is chosen the same for all image channels /;, ensuring 
that channels are smoothed with a coherent multivalued geometry which takes the 
correlation between channels into account (since T depends on G). Equation (69) 
unifies a lot of existing scalar or multivalued regularization approaches and proposes 
at the same time two interpretation levels of the regularization process: 


¢ Local interpretation: Equation (69) may be seen as the physical law describing 
local diffusion processes of the pixels individually regarded as temperatures or 
chemical concentrations in an anisotropic environment which is locally described 
by T. 

¢ Global interpretation: The problem of image regularization can be regarded as 
the minimization of the energy functional FE (1) by a gradient descent (i.e., a 
PDE), coming from the Euler-Lagrange equations of FI) [3, 19,49, 51,78]: 


EM = [ varayaa where wy : R?> R (70) 
Q 


* Itresults in a particular case of the PDE (69), with T = Bt g-9-* + oor ett 
where A,,A_ are the two positive eigenvalues of the non-smoothed structure 


tensor field G = )°; VJ; V1? and 64, 6_ are the corresponding eigenvectors. 


Unfortunately, there are local configurations where the PDE (69) does not fully 
respect the geometry T and where the smoothing is performed in unexpected direc- 


tions. For instance, considering (69) with tensor fields Ti(x) = (ivi) (er) 
(purely anisotropic) and Ta.) = Ig (purely isotropic) lead both to the heat 
equation a = AT which has obviously an isotropic smoothing behavior. Different 
tensors fields T with different shapes (isotropic or anisotropic) may define the 


same regularization behavior. This is due to the fact that the divergence implicitly 
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introduces a dependance on the spatial variations of the tensor field T, so it hampers 
the design of a pointwise smoothing behavior. 


Trace-Based PDEs 

Alternative PDE-based regularization approaches have been proposed in [3, 51, 
68, 69, 78] in order to smooth an image directed by a local smoothing geometry. 
They are inspirit very similar to the divergence equation (69), but based on a trace 
operator: 


al eh Fh 

. i ; : —_ dx2 axdy 

Vi= Petey, a = trace (TH; ) with H; = ah ah (71) 
axdy dy? 


H; stands for the Hessian of /;. Equation (71) is in fact nothing more than a 


: alg + 
tensor-based expression of the PDE a = to aty Ip-g- + fa-aty Ip+9+ where 
2 . ; : a: 
Ig-p- = ot. This PDE can be viewed as a simultaneous combination of two 


orthogonally oriented and weighted 1D Laplacians. In case of multivalued images, 
each channel J; of I is here also coherently smoothed with the same tensor field 
T. As demonstrated in [78], the evolution of Eq. (71) has a geometric meaning in 
terms of local linear filtering: it may be seen locally as the application of very small 
convolutions around each point X with a Gaussian mask cI oriented by the tensor 
Ty): 


1 X? TX 
T _ 
ae ra (- At ) 


This ensures that the smoothing performed by (71) is indeed oriented along the 
predefined smoothing geometry T. As the trace is not a differential operator, 
the spatial variation of T does not trouble the diffusion directions here and two 
different tensor fields will necessarily lead to different smoothing behaviors. Under 
certain conditions, the divergence PDE (69) may be also developed as a trace 
formulation (71). But in this case, the tensors inside the trace and the divergence are 
not the same [78]. Note that trace-based equations (71) are rather directly connected 
to functional minimizations, especially when considering the multivalued case. For 
scalar-valued images (n = 1), some correspondences are known anyway [3, 19,51]. 


Curvature-Preserving PDEs 

Basically, the divergence and trace Eqs. (69) and (71) locally behave as oriented 
Gaussian smoothing whose strengths and orientations are directly related to the 
tensors T,,). But on curved structures (like corners), this behavior is not desirable: 
in case of high variations of the edge orientation 0~, such a smoothing will tend 
to round corners, even by conducting it only along 0~ (an oriented Gaussian is not 
curved by itself). To avoid this over-smoothing effect, regularization PDEs may try 
to stop their action on corners (by vanishing tensors T,,) there, i.e., f~ = f+ = 0), 
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but this implies the detection of curved structures on images that are themselves 
noisy or corrupted. This is generally a hard task. 

To overcome this problem, curvature-preserving regularization PDEs have been 
introduced in [77]. We illustrate the general idea of these equations by considering 
the simplest case of image smoothing along a single direction, i.e., a vector field 
w : 2 — R?’ instead of a tensor-valued one T. The two spatial components of w are 
denoted Wx) = (Uy V(x) - 

The curvature-preserving regularization PDE that smoothes I along w is defined 
as 


du du 

Vi =1,...,n, a = trace (ww’ H;) + VI) Jww with Jy = ( 4 

(72) 

where Jy stands for the Jacobian of w. Equation (72) simply adds a term VJ TIww 

to the corresponding trace-based PDE (71) that would smooth I along w. This 

term naturally depends on the variation of the vector field w. Actually, it has been 

demonstrated in [77] that Eq.(72) is equivalent to the application of this one- 
dimensional PDE flow: 


x 
a1;(Ciay) 0 L;(Ca)) Co = X 
= th x 73 
at da ae eo w (ck) i) 


where C ey is the streamline curve of w, starting from X and parameterized by a € R. 
Thus, Eq. (73) is nothing more than the one-dimensional heat flow constrained on 
the streamline curve C. This is indeed very different from a heat flow oriented by 
w, as in the formulation uh = ua since the curvatures of the streamline of w are 
now implicitly taken into account. In particular, Eq. (73) has the interesting property 
to vanish when the image intensities are constant on the streamline CX, whatever 
the curvature of CX is. So, defining a field w that is tangent everywhere to the 
image structures allows the preservation of these structures during the regularization 
process, even if they are curved (such as corners). 

Moreover, as Eq. (73) is a 1D heat flow on a streamline CX, its solution at time 
dt can be estimated by convolving the image signal lying on the streamline CX by a 
1D Gaussian kernel [50]: 


wxea, rl [yea (c%,) G"(p) d (74) 
, wo] () ee a 


This formulation is very close to the line integral convolution (LIC) framework [17], 
which has been introduced as a visualization technique to render a textured image 
representing a 2D vector field w. As we are considering diffusion equations here, the 
weighting function in Eq. (74) is naturally Gaussian. This geometric interpretation 
particularly allows to implement curvature-preserving PDEs (74) using Runge- 
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Fig. 27 Streamline C* of various vector fields w:2 — R?. (a) Streamline of a general field w. 
(b) Example of streamlines when w is the lowest eigenvector of the smoothed structure tensor G, 
(one block is one color pixel) 


Kutta estimations of the streamline geometries, leading to sub-pixel precision of 
the smoothing process. 

This single-direction smoothing PDE (72) can be easily extended to deal with 
a tensor-valued geometry T: £2 —> P(2), in order to be able to represent both 
anisotropic or isotropic regularization behaviors. This is done by decomposing 
the tensor field T as the sum of several single-directional tensors, ie., T = 


T 
2 et (VTa.) (VTa.) da, where a, = cosa sina’. This naturally suggests 
to decompose a tensor-driven regularization process into a sum of single-direction 
smoothing processes, each of them being expressed as a curvature-preserving PDE. 
As a result, the corresponding curvature-preserving PDE directed by a tensor field 


T 1S 
Vi = | neo L —— = trace TH; + V I VTa da 75 
2 yews R) ( ) I J ta, a ( ) 


When T is locally isotropic (on homogeneous region), Eq. (75) is similar to a 2D 
heat flow, while when T is locally anistropic (on an image contour), it behaves as a 
1D heat flow on the streamline curve following the contour path, thus taking care of 
its curvature (Fig. 27). 


Applications 


Some application results are presented here, mainly based on the use of the 
curvature-preserving PDEs (75). A specific diffusion tensor field T has been used to 
adapt the smoothing behavior to each targeted application. 
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Noisy color image (/eft), denoised image (right) by 
curvature-preserving PDE (50.75) 


After several iterations After several iterations of 
of trace-based PDE (50.71) curvature-preserving PDE 
(50.75) (with same tensor 
field T) 


Fig. 28 Using PDE-based smoothing to regularize color and grayscale images 
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Original color image (/eft), image with 50% pixel removed (middle), reconstructed 
using PDE (50.75) (right) 


a2 eatin # 
x 


Original color image (/eft), reconstructed using PDE (50.75) (right) 
(the in painting mask covers the cage) 


Fig. 29 Image inpainting using PDE-based regularization techniques 


Color Image Denoising 

Image denoising is a direct application of regularization methods. Sensor inaccura- 
cies, digital quantifications, or compression artifacts are indeed some of the various 
noise sources that can affect a digital image, and suppressing them is a desirable 
goal. Figure 28 illustrates how curvature-preserving PDEs (75) can be successfully 
applied to remove such noise artifacts while preserving the thin structures of the 
processed images. The tensor field T is chosen as in Eq. (68). 


Color Image Inpainting 

Image inpainting consists in filling in missing (user-defined) image regions by 
guessing pixel values such that the reconstructed image still looks natural. Basically, 
the user provides one color image I : 2 —> R? and one mask image M: 2 — {0, 1}. 
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The inpainting algorithm must fill in the regions where M(X) = 1, by means of 
some intelligent interpolations. Image inpainting using diffusion PDEs has been 
proposed, for instance, in [10, 18, 78]. Inpainting is a direct application of our 
proposed curvature-preserving PDE (75), where the diffusion equation is applied 
only on the regions to inpaint, allowing the neighbor pixels to diffuse inside these 
regions in an anisotropic way (Fig. 29). 


Visualization of Vector and Tensor Fields 

Regularization PDEs such as (69), (71), and (75) can be also used to visualize 
a vector field w : 2 — R? ora tensor field G: Q > P(2); see also Sect. 5. 
The idea is to smooth an originally pure noisy image using a diffusion tensor 
field T which is chosen to be T = ww’ or T = G or other variations as long 
as the smoothing geometry is indeed directed by the field we want to visualize. 
Whereas the PDE evolution time ¢ goes by, more global structures of the considered 
fields appear, i.e., a visualization scale-space is constructed. The same PDE-based 
visualization technique allows to display interesting global rendering of DT-MRI 
volumes (medical imaging) displaying “stuffed” views of the fiber map (Fig. 30). 


7 Conclusion 


This chapter presented a selection of possible approaches for systematic treatment of 
multidimensional data sets and algorithms based on differential methods. Such data 
sets may be the result of some image processing, for instance, a three-dimensional 
stack of CT slices in medical imaging used to extract a triangular surface represent- 
ing bones. Visual data analysis is particularly important for big data. Many such 
data sources originate in computational sciences, produced by algorithms based 
on differential methods. Utilizing the same concepts for image processing and 
multidimensional data in general allows generalization of methods and increased 
software reusability and applicability. Section 2 reviews a mathematically founded 
model to structure multidimensional data covering a wide category of data types. 
Since there is no commonly agreed standard in the scientific community on how to 
lay out multidimensional data for computational purposes, a multitude of alternative 
models exist. For a specific application, it is subject of investigation whether some 
specific model would fit all the respective requirements. Section 3 delves deeper 
into the modeling of mathematical operators using computational data structures. A 
mathematical algorithm must be cast into a discretized form in order to be applicable 
in a numerical code. Differential forms are a mathematical abstraction allowing 
coordinate-free formulations of partial differential equations, which are the basis 
of many physical and engineering methods, as well as image processing. Geometric 
Algebra, reviewed in Sect. 4, extends the notions of the commonly known vector 
algebra to form a full system of algebraic operations to include (among others) 
the notion of “dividing vectors.” While unfamiliar at first, the result is a visually 
intuitive way to phrase complex algebraic operations, enabling better insight and 
more efficient implementation as compared to “ad hoc” approaches. An important 
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Vector field visualization Visualization using PDE Visualization using PDE 
with arrows (50.75) (after 5 iter) (50.75) (after 15 iter) 


Tensor field displayed with ellipsoids Tensor field rendered using a PDE approach 
(left) and tracked fibers (right) (50.75) 


Fig. 30 Visualization of vector and tensor fields using PDEs 


application of differential methods for multidimensional data is the investigation of 
features in vector fields. While primarily of interest to computational sciences such 
as computational fluid dynamics, identifying topological features in vector fields 
may well apply to color images with RGB channels and gradients of grayscale 
images or even more to stacks of images such as three-dimensional data and 
animation sequences. Section 5 presents some basics for feature detections in such 
data sets. While the application to animation sequences is beyond the scope of 
this chapter, utilizing the topological skeleton of some data set may well have 
potential for data compression or automatizing algorithms for motion pictures. 
Partial differential equations (PDEs) furthermore allow for direct improvement of 
image quality and feature reconstruction, as explained in the algorithms presented 
in Sect. 6. PDEs describing diffusion are particularly suitable for reducing noise in 
images, recovering lost features, as well as the direct visualization of vector and 
tensor fields. The set of presented algorithms in this chapter is still subject of active 
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research and neither a comprehensive nor the only reasonable approach; rather, the 
various aspects covered provide inspirations covering many scientific disciplines 
under one hood. 


Cross-References 


Tomography 
Mathematical Methods in PET and SPECT Imaging 
Mathematics of Electron Tomography 
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Autocorrelation, 772 

Autocovariance function (ACF), 1378, 1379 

Automated image registration (AIR), 152 
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Backprojection, 782 

Bacon, Francis, 8 

Bag of features (BOF), 1901 

Balls phantom, 1016-1020 

Banach spaces, 98-104, 274 

Banach’s theorem, 1719 

Band extraction, 2080 

Band-limited signals, 19-20 

Bandwidth, 776 

Basis, 1721-1722 

Basis functions, 813 

Bayes estimation, 329-330 

Bayes’ formula, 1369 

Bayes risk, 1441 

Bayes’ rule, 314 

Bayesian approach, 2086 

Bayesian estimate, 818 

Bayesian framework 
approximation error approach, 1065-1067 
for inverse problem, 1061-1062 
inference, 1062 
likelihood and prior models, 1062-1063 
nonstationary problems, 1063-1065 

Bayesian inference, level set based tracking, 

1932-1933 
Beer—Lambert law, 1035, 1036 
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Bellman optimality principle, 1874 
Beltrami equation, 728 
Bernoulli matrices, 221—222 
Besov norms, 1010 
Bessel bound, 1720 
Bessel function, 1146 
Bessel sequence, 1720-1721 
Best k-term approximation, 213-214 
Bilateral filter, 1602, 1603, 1647-1658 
Blagovestchenskii identity, 1226, 1227 
Blind deconvolution, 52, 1360-1361, 
2089-2090 
Blob, 814 
Block iterative ART, 1004 
Bojarski identity, 686 
Boltzmann-Shannon entropy, 270, 282 
Born approximation, 661, 685-686, 769, 
793-794, 1048-1049, 1179-1181 
error estimate, 1258-1261 
explicit formula for slab, 1255-1258 
modified, 1194 
Bouguer’s law, 14 
Boundary control method, 1207-1208, 
1211-1234, 1273-1275 
Boundary derivative operator, 1045 
Boundary distance function, 1209 
Boundary rigidity problem, 1209 
Bregman iteration, 1521-1524 
Burg’s entropy, 365 


Cc 

Calderén’s technique, 717 

Canonical dual frame, 1723 

Cardinal B-splines, 1677, 1681 

Cartesian products, 1346 

Cauchy data, 720-721 

Cauchy—Green deformation tensor, 1831 

Cauchy strain tensor, 2013 

Cauchy stress law, 1827 

Caustics and trapped rays, 1270-1271 

C-band, 765 

Census transform, 1983 

Centroidal Voronoi sampling, 1871 

Centroidal Voronoi tessellation (CVT), 
1871 

Chains, 2122-2124 

Chambolle’s algorithm, 1473-1474 

Chambolle’s dual method, 1516-1518 

Chan-Golub-Mulet’s primal-dual method, 
1516 

Chan—Vese model, 483 

Charged Coupled Device (CCD), 963 

x°-Divergences, 138-140 
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Chunking, 1446 
Cimmino—Landweber methods, 374-377 
Circular integrating detectors, 1125 
Closed-form inversion formulas, 1149-1151 
Coarea formula, 1461 
Coarse-to-fine strategy, 1969 
Cochains, 2124-2128 
Coherence, 220-221, 782 
Coherent interferometric (CINT) imaging 
active arrays, 1331-1335 
cross-correlations, 1321 
mean point spread function, 1324-1326 
SNR, 1322-1324, 1326-1327 
Wigner transform, 1327-1331 
Coherent multi-valued geometry, 2151 
Cohomology, 2128-2130 
Collimated source model, 1042-1043 
Color image recognition, 1427-1428 
Color image restoration, 1577-1579 
Color level set technique 
binary level set model, 484 
of multiple shapes, 484 
for reservoir characterization, 486-487 
for tumor detection, 485-489 
Combined phase and amplitude contrast 
model, 945 
Complete electrode model (CEM), 711, 746 
Complete orthonormal system, 23 
Complex geometrical optics (CGO), 717-720, 
724 
Complimentary slackness conditions, 288 
Compressive sensing (CS), 206-248, 
1992-1994 
Computational anatomy, 1762 
Computational efficiency, Geometric Algebra 
using Gaalop, 2139-2141 
Computerized tomography (CT), 232, 802, 
804, 805 
Conditional covariance, 1062 
Conditional density, 1347 
Conditionally Gaussian hypermodels, 
1361-1362 
Conditional mean (CM), 1062, 1366-1368 
Conditional probability, 1061 
Conductivity distribution, 489 
Cone beam CT, 867-868 
Conformal Geometric Algebra, 2136-2139 
Congruences, 1864 
Conical tilt electron microscope tomography, 
microlocal analysis, 889-891 
Conical tilt ET, 868-871 
Conjugate gradients, 837 
Continuity results, X-ray transform, 
863-864 
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Continuous case 
acceptable preferred data, 400-401 
missing data, 401-402 
preferred data selection, 401 
Continuous case with Y = h(X) 
censored exponential data, 403-405 
example, 402-403 
general approach, 405 
Continuous-to-discrete (C-D) imaging models, 
1105-1106 
Contrast transfer function (CTF), 961 
Convergence properties, 238-240 
Convex analysis, 281-293 
Convex Mean Value Theorem, 278 
Convolution operator, 346 
Convolution theorem, 18, 1727 
Correlation, 772 
Correlation coefficient, 131 
Correspondence problem, 1947-1949 
Cost function, 490, 1440 
Co-tangential space, 2104 
Covariance operator, 1840 
Co-vectors, 2104-2105 
Covering number, 1443 
Cowpea mosaic virus (CPMV), 1020 
Crack detection, 479 
Cramér—Rao lower bound, 1364 
Cross-entropy (CE) problems, 363-366 
Cross-range resolution, 785-786 
Cryo-fixation method, 942 
Csiszar Divergences, 137-141 
Curvature-preserving PDE’s, 2152-2156 
Curvelet decomposition, 1210-1214, 1239 
Cut locus distance function, 1216 
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Data, NGC 2997, 2065 

Data access ordering, 818, 832 
Data acquisition geometry, 971-973 
Data-collection manifold, 776 

Data compatibility, 101 

Data discrepancy, 976-977 

Data model, 2102 

Dr. David E. Kuhl, 907 
DC-programming, 1980 
Deblending, 2059, 2078 
Deblurring, 1349-1350 

Decoder, 224 

Decomposition methods, 675-677 
Deconvolution, 50-52, 266-268, 2081-2094 
Delaunay mesh, 1872 

Denoising, 1349, 1607-1614, 1656 
Density estimation, 133-137 
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Derivative-of-Gaussian (DoG) filters, 1954 
Detector model, 982 
Deterministic pixel measure, 127 
Dichotomy class divergences, 139 
Dielectric medium, 1172 
Diffeomorphism, 736, 1777-1780, 1798-1800 
Difference imaging, 747 
Differentiable manifold, 1864 
Differential forms, 2119-2122 
Differential motion approach, 1950-1956 
Differential motion estimation, 1951-1954 
Differential path length factor (DPF), 1036 
Diffuse source model, 1043 
Diffusion approximation (DA) 
Boltzmann hierarchy approach, 1040 
boundary conditions, 1041-1042 
collimated source model, 1042 
Monte Carlo diffusion approach, 1044 
numerical solution methods, 1043 
photon density and photon current, 1040 
validity, 1043 
Diffusion-based optical tomography (DOT), 
1047 
Diffusion geometry, 1866-1868 
Diffusion kernel, 1866-1868 
Diffusion tensors images, 1513-1514 
Digital difference analyzer (DDA), 821 
Dijkstra’s algorithm, 1874-1875 
Dirac delta distribution, 502 
Direct regularization methods, 93-107 
Direct scattering problems, 652-653 
Helmholtz equation, 654-657 
Maxwell equations, 661-666 
obstacle scattering, 657-660 
Dirichlet boundary value problems, 1222 
Dirichlet-to-Neumann map, 114, 703, 
720-728, 732-734, 739-741 
Discrepancy principle, 435 
Discrete Fourier transform (DFT), 1735 
Discrete geodesics, 1846-1848 
Discrete histogram, 131 
Discrete modulation, 1735 
Discrete path length, 1845 
Discrete Picard condition, 57 
Discrete reconstruction problem, 816 
Discrete-to-discrete (D-D) imaging models, 
1108-1109 
Discrete total variation, 1477-1479, 
1505-1507 
Discretized Laplace—Beltrami operator, 1880 
Distance functions, 1949 
Distorted wave Born approximation, 957 
Domain inpainting, 1508 
Dose problem, 977-978 
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Down-range resolution, 785 

Dual and primal-dual methods, 1515-1516 

Dual decomposition, 1891 

Duality gap, 289 

Duality operator, 1774 

Dual problem, 1439 

Dyadic decomposition, 1240 

Dynamic imaging, 1053 

Dynamical shape priors, variational 
segmentation, 1935-1936 

Dyson equation, 1049 


E 
Edge linking model, 1485 
Effective dipole method, 543 
Efficient ordering, 832 
Eikonal equation, 1235-1236, 1875-1876 
Einstein ring, 11 
Ekeland’s variational principle, 291 
Elasticity-based PCA, 1837-1843 
Elastic registration, 149 
Elastic regularization functionals, 2012-2014 
Elastic scattering, 947 
Elastic shape average, 1835-1837 
Electrical impedance tomography 
bibliography and open questions, 544-545 
mathematical model, 538-539 
numerical methods, anomaly detection, 
541-544 
physical principles, 537-538 
voltage perturbations, asymptotic analysis, 
539-541 
Electromagnetic waves scattering, 766 
Electron A-tomography (ELT), 994-998 
Electron micrograph, 805 
Electron microscope tomography (ET), 
853-856 
Electron microscopy, 805 
EM algorithm, 339-341 
macromolecular assemblies, 337 
maximum likelihood problem, 337-339 
weighted least-squares problem, 342 
Electron—specimen interaction, 946-947 
Electron wave, 947 
Electrosensing, 702 
Electrostatic potential models, 950 
Elliptic equation, 114-116 
Embedded manifolds, 1865 
Emission computed tomography (ECT), 185 
Emission tomography, 330-337 
Gibbs smoothing, 347-348 
Good’s roughness penalization, 345-347 
Poisson random variable, 331-332 
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regularization, need for, 342-343 
Shepp-Vardi EM algorithm, 332-337 
smoothed EM algorithms, 343-344 
Empirical risk minimization (ERM), 1442 
Encoder, 224 
Energy function, 1963 
Entropy regularization, 1006-1008 
Epipolar constraint, 1974 
Euclidean geometry, 1864 
Euler equations, 345 
Euler-Lagrange equations, 1552, 1559, 1566 
Euler—Poincaré equation, 1776-1777 
Evolution equations, 1802 
Exact penalization veto, 101 
Expectation—maximization (EM) algorithms, 
460-464, 1371-1374 
ART and Cimmino—Landweber methods, 
374-377 
continuous case, 399-405 
convergence, 426 
Csiszar and Tusnady approach, 409-412 
data binning, 323-327 
deconvolution problem, 319-327 
discrete case, 397-398 
empirical Bayes estimation, 329-330 
finite mixture problems, 423-426 
finite mixtures, 407 
Gibbs smoothing, 347-348 
Good’s roughness penalization, 345-347 
and Kullback-Leibler distance, 407-409 
MART and SMART methods, 377-380 
maximum-likelihood problem, 311, 317 
minimum cross-entropy problems, 
363-366 
missing data, 398 
monotonicity properties, see Monotonicity 
properties 
multinomial distribution, 414 
multinomial example, 406 
multiplicative iterative algorithms, 
369-370 
multiplicative smoothing, 344 
non-negative solutions, for linear equations, 
417-423 
nonnegative least squares, 366-369 
NSEM, 391-393 
ordered subset EM algorithm, 371-374 
Poisson sums, 412 
Poisson sums ECT reconstruction problem, 
415-416 
Poisson sums using KL distance, 417 
Radon-Nikodym derivatives, 312 
row-action and block-iterative EM 
algorithms, 380-384 
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Shepp-Vardi EM algorithm, 332-337 
smoothing operator, 343 
stochastic EM algorithm, 393-396 
Explicit vs. implicit shape representation, 
1911-1913 
Exponential charts, 1768 
Extended Kalman filtering (EKF), 1064 
Extension, movies, 1614 
Exterior algebra, 2108 
Exterior product, 2107 
Extrapolation of band-limited signal, 19-20 


F 
Factorization method, 594-595, 654, 692-693 
conducting obstacles, 603-610 
crack problem, 614-615 
half space problem, 612-614 
impedance tomography, insulating 
inclusions, 595—603 
inverse acoustic scattering, sound-soft 
obstacle, 616-622 
inverse electromagnetic scattering, 
inhomogeneous medium, 622-627 
local data, 611-612 
Fan beam parameterization, 868 
Far field approximation, 1179-1181 
Far field operator, 269, 682, 686-688 
Far field pattern, 653 
Farthest point sampling (FPS), 1870-1871 
Fast marching methods, 1876 
f-Divergences, 137-141 
Feature-based methods, 1899-1902 
Feature extraction, 1394 
Fenchel conjugate, 260, 281-284, 1439 
Féjér monotonicity, 349 
Fenchel duality, 284-285 
Fermat’s rule, 277 
Fermi—Dirac entropy, 282 
Fiber-bundle classification scheme, 2114 
Fick’s law, 1040-1042 
Field properties, 2116-2117 
Figure of merit (FOM), 827 
Filter, 787 
Filtered backprojection (FBP), 782, 787, 811, 
864-865, 906, 990-994, 1149 
Filtering, 2078 
f -Information, 141-146 
Finite element method (FEM), 744, 1043 
Finite mixture problems 
acceptable data, 424 
convergence of Mix-EM algorithm, 426 
likelihood function, 423 
Mix-EM algorithm, 425-426 
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motivating illustration, 424 
probability density functions, 423 
Finite time response operator, 1213 
First order Born approximation, 956 
First order pseudo-differential operator, 709 
Fisher information matrix, 1358-1359 
Fixed point theory, 292-293 
Fleishman’s bilateral filter, 1650 
FLIRT, 151 
Fluence rate perturbations, 584 
Focus of expansion (FOE), 1959-1960 
Footprint, 779, 780 
FORBILD thorax phantom, 839 
Formal adjoint, 781 
Forward mapping, 1134-1135 
Forward model, 944-971 
Forward operator, 474, 491, 966-970 
Fourier integral operators, 880-882 
Fourier transform, 346, 1092, 1093, 1724-1726 
Fréchet derivative, 500 
Fréchet distance, 1789 
Fragment, 2119 
Frame, 1722-1724 
Frame bounds, 1722-1724 
Frame coefficients, 1723 
Frame decomposition, 1723 
Frame operator, 1723 
Fredholm integral equations, 269-271, 
297-298 
Free surface boundary condition, 1039 
Frequency 
bands, 765 
domain, nonlinear problem in, 1265-1266 
shifts, 1727 
Functional photoacoustic tomography, 
1088-1089 
Function spaces, 859-860 
Fundamental solution, 766 
Fuzzy C-means, 1650 


G 

Gabor analysis, 1718 
Bessel sequences, Hilbert spaces, 

1720-1721 

convolution, involution and reflection, 1727 
discrete gabor systems, 1734-1738 
Fourier transform, 1724-1726 
frame bounds, 1722 
frame decomposition, 1723 
non-separable atoms, 1751-1752 
orthonormal basis, 1721-1722 
pseudo-inverse operator, 1719-1720 
sampled STFT, 1746-1747 
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Gabor analysis (cont.) 
STFT, 1727-1730 
tight frames, 1724 
time-frequency shifts, 1731, 1733 
translation and modulation, 1726-1727 
Gabor atom, 1732 
Gabor frame theory 
finite discrete periodic signals, 1735-1736 
in €2(Z), 1734-1735 
in C4, 1736-1738 
in L?(R¢), 1730-1734 
Galilei, Galileo, 7-8 
Gaussian beams, 1230-1234 
Gaussian low-pass filter, 1954 
Gaussian matrices, 221 
Gaussian noise, 1356 
Gaussian point spread function, 17, 19 
Gauss map, 128 
Gauss-Newton method, 68, 69 
Gelfand widths, 210-211, 224-227 
Generalized cross validation (GCV), 66 
Generalized Kaiser-Bessel window function, 
814 
Generalized Kullback—Leibler (KL) 
divergence, 2085 
Generalized multidimensional (GMDS) 
scaling, 1889-1891 
Generalized portrait algorithm, 1398 
Geodesic active contour model, 1484-1486 
Geodesic equation, 1766-1767 
Geodesic path, 1829 
Geodesics, 1865 
Geometric Algebra, 2109-2110, 2132 
Geometrical optics, 954-955, 1237-1238 
Geometric realization, 1872 
Global minimizers, 170 
Global point signature (GPS), 1888 
Global stability, 730-731 
Golub-Kahan bidiagonalization (GKB), 62 
Gradient descent evolution, kernel density 
estimator, 1929-1930 
Gradient direction, 501 
Gradient methods, 435 
Grady segmentation method, 1635 
Graph cut methods, 1524 
Green’s function, 767, 1045-1046, 1048, 1049, 
1152 
Green’s operator, 1055 
Green-St.-Venant strain tensor, 2013 
Grenander’s theory of deformable templates, 
1762 
Gromov—Hausdorff distance, 1822, 1888-1891 
Gromov—Wasserstein distance, 1892 
Group testing, 211 
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H 
Hahn-Banach/Fenchel duality circle, 275 
Hahn—Banach theorem, 275-276 
Half-time reconstruction problem, 1102-1104 
Hamilton flow, 1244 
Hamiltonian approach, 1772-1773 
Hamilton—Jacobi-type equation, 497 
Hammersley—Clifford theorem, 1353 
Hand-written digit recognition, 1425-1427 
Hankel transform, 1146 
Hard margin classifier 
linear learning, 1400-1402 
nonlinear learning, 1413-1414 
Hard margin regression nonlinear learning, 
1414-1415 
Hausdorff distance, 1789, 1883-1884 
Hausdorff measure, 1548 
Hausdorff topology, 2112 
Heating function, 1084 
Heat kernel, 1867 
Heat kernel signature (HKS), 1900 
Heat operator, 1868 
Heaviside function, 499 
Helical CT, 805 
Helmholtz equation, 654-657, 1147 
Henyey-—Greenstein scattering function, 1038, 
1040 
Herglotz wave function, 677 
Hessian matrix, 81 
High-range-resolution (HRR) pulse, 773 
Hilbert spaces, 96-98, 104-106, 1763 
Hinge loss function, 1403 
Histogram, 2119 
concentration, 1635-1639 
Hoffman phantom, 924 
Hélder regularity, 1467-1468 
Holder stability, 1133-1135 
Homeomorphism, 1217-1221 
Homogeneous coordinates, 1956 
Homology, 2128-2130 
Horizontality condition, 1772 
Horn and Schunck approach, 1964-1967 
Huygens’ principle, 659, 677-682 
Hybrid method, 678 
Hyperelastic regularization functionals, 
2014-2015 
Hyperelastic regularizer, 2015 
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Illumination, 945-946 
Image, 50 
deblurring, 50-52 
denoising, 266-268 
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dose, 983 
formation, 1349 
kernel, 784 
mode, TEM, 941-942 
motion analysis, 1946 
processing, 1676, 1701, 1704, 1705 
reconstruction, 472-585, 811, 904 
registration, 126, 148-149 
restoration, 1488-1495 
restoration with impulsive noise, 
1573-1577 
segmentation, 1511-1513, 1910 
segmentation via Bayesian inference, 
1913-1915 
Image quality (IQ) phantom, 923 
Impediography 
bibliography and open questions, 565 
mathematical model, 563-564 
physical principles, 561-562 
substitution algorithm, 564-565 
Implicit shapes 
dynamical shape priors, 1931-1936 
linear dynamical models, 1934 
Impulse response, 784 
Incident field, 767 
Inelastic electron scattering, 949-952 
Inexact matching, 1810-1811 
Inexact Newton methods, 73 
Infimal convolution, 283 
Infrared thermal imaging 
asymptotic analysis, temperature 
perturbations, 555-557 
bibliography and open questions, 560 
numerical methods, 549-554 
physical principles, 555 
Inhomogeneous coordinates, 1956 
Inhomogeneous medium, scattering, 660-661, 
669-671 
Initial-boundary value (IBV) problem, 1126 
Initial convex sequence estimator (ICSE), 1381 
Initial monotone sequence estimator (IMSE), 
1381 
Initial positive sequence estimator (IPSE), 
1380 
Inpainting, 1349 
Integrated autocorrelation time (IACT), 1380 
Integrated data function, 1092 
Intensity-based image registration technique, 
150 
Intensity operator, 962 
Interpolation by radial basis function, 
1417-1418 
Interscale relation, 2077 
Intrinsic alignment 
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invariance, 1923-1924 
translation invariance, 1924-1925 
Intrinsic symmetry, 1897-1898 
Invariance, 1947-1949 
Invariant shape similarity, 1861, 1882-1883 
canonical forms, 1885-1888 
graph-based methods, 1891 
Gromov—Hausdorff distance, 1888-1891 
Gromov—Wasserstein distances, 1892-1893 
rigid similarity, 1883-1885 
shape DNA, 1893 
Inverse, 811 
Inverse attenuated Radon transform (IART), 
906, 910-911, 917-918 
Inverse crime, 1383 
Inverse gamma distribution, 1361 
Inverse kinematic problem, 1209 
Inverse medium problem, Newton iterations, 
682-683 
Inverse problems, 977-985 
Inverse Radon transform, 906, 913-917 
Inverse scattering, 268-269, 296-297, 1275 
iterative and decomposition methods, 
672-686 
problem, 652 
qualitative methods, 686-696 
uniqueness, 667-671 
Inverse synthetic-aperture radar (ISAR), 
774-7175 
resolution, 784-785 
Involution, 1727 
Isometric embedding, 1869 
Isometry, 1863 
Isotropic undecimated wavelet transform 
(IUWT), 2062 
Iteration operator, 313, 353 
Iterative algebraic techniques, 1153 
Iterative closest point (ICP) algorithms, 1884 
Iterative image reconstruction, 1110-1113 
Iteratively regularized Gauss-Newton method, 
451-455 
Iteratively reweighted least squares (IRLS), 
236-243 
Iterative methods, 1001-1005, 1471-1477 
Iterative regularization, 37-39, 61, 435 
Iterative step, 818 
ITK, 151 


J 

Jacobian matrix, 747 

Jensen-Shannon divergence, 138 

Joint restoration and segmentation, 
1582-1584 
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K 
Ka-band, 765 
Kaczmarz method 
in frequency domain, 1266 
time domain, 1261-1263 
Kaczmarz method time domain, 1261-1263 
Kaczmarz’s algorithm, 32 
Kaczmarz-type methods, 458-460 
Kalman filtering, 1064 
Karcher mean, 1803 
Kendall’s theory, 1760 
Kernel density estimation, 134-135 
level set domain, 1926-1929 
Kernel-trick, 1412-1413 
Kirchhoff imaging, 554 
Kirchhoff/physical optics approximation, 
659 
Kirchhoff—Poisson formulas, 1143 
Kirchhoff-type imaging, broad range of 
frequencies, 551-552 
Kriging, 1415-1419 
Ku-band, 765 
Kuhn-Tucker conditions, 1438 
Kullback—Leibler divergence, 138, 
309-310, 315 


L 

Lagrange multipliers, 289-291, 1437 

Lagrangian duality, 287-289 

Landweber iteration, 39 

Landweber method, 61 

Laplace—Beltrami operator, 734, 1865 

Laplacian field, 110 

Laplacian kernels, 1771 

Laplacian operator, 1083 

Large deformation diffeomorphic metric 
mapping (LDDM), 1013 

Large deformation diffeomorphic metric 
matching (LDDMM) approach, 1987 

Laser-based photoacoustic tomography, 
1086-1087 

Layered medium, 1193-1194 

L-band, 765 

L-curve, 67 

Least squares cost functionals, 500-501 

Least squares methods, inverse medium 
problem, 683-685 

Lebesgue-space, 1821 

Lens-less imaging, 960 

Lens rigidity problem, 1210 

Level set methods, 1851 

binary media, 479-48 1 
color level set, see Color level set technique 
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cost functionals, 490 
cracks/thin shapes, 487-489 
Eulerian derivatives, 492-493 
geometric quantities, 495-497 
material derivative method, 493-494 
piecewise constant level set function, 
482-483 
single smooth level set function, 482 
transformations and velocity flows, 
491-492 
vector level set, 483 
Levenberg—Marquardt method, 447-451 
Likelihood density, 1348 
Likelihood distribution, 1061 
Limited angle Lambda CT, 867 
Limited data X-ray CT, 887-888 
Linear approximations, 1052-1053 
Linear detectors, 1124-1125 
Linear diffusion, 1634-1639 
Linear discriminants, 1398 
Linear Gaussian shape priors, 1916-1920 
Linear hard margin support vector classifier, 
1398 
Linear inverse problems, 262-263, 
293-295 
alternating projection theorem, 31-32 
band-limited signals, 19-20 
compact operators and SVD, 27-30 
Cormack’s inverse problem, 14-16 
deblurring problem, 17-19 
discretization, 39-43 
forward and reverse diffusion, 16-17 
iterative regularization, 37-39 
linear operators, 25-27 
Moore-Penrose inverse, 30-31 
PET, 20-21 
Platonic inverse problem, 11-14 
renaissance, 7 
Tikhonov regularization, 32-37 
weak convergence, 23-25 
Linear learning, 1398-1410 
Linear least squares approach, 
1396 
Linear sampling method, 593, 628-631, 653, 
688-691 
Lippmann-Schwinger equation, 660, 1178 
frequency domain, 768 
integral equation, 654, 768 
Lipschitz properties, 272-273 
Lipschitz stability, 1133-1135 
Lloyd—Max algorithm, 1871 
Local energy decay estimates, 1132 
Local minimizers, 168-170 
Logistic loss functions, 1407 


Index 


M 
Magnetic induction tomography (MIT), 706 
Magnetic resonance elastography 
asymptotic analysis, displacement fields, 
575-577 
bibliography and open questions, 579-580 
mathematical model, 573-575 
numerical methods, 578-579 
physical principles, 573 
Magneto-acoustic imaging 
magnetic resonance elastography, 573-580 
magneto-acousto-electrical tomography, 
567-570 
photo-acoustic imaging of small absorbers, 
580-584 
with magnetic induction, 570-572 
Magneto-acousto-electrical tomography 
mathematical model, 567-568 
physical principles, 567-570 
substitution algorithm, 568-570 
Majorization-minimization (MM), 1530-1532 
Marginal densities, 1346 
Marginal polytope, 1986 
Markov chain, 1376 
Markov Chain Monte Carlo (MCMC) 
sampling, 1374-1382 
Markov model, 1351 
Markov random field (MRF), 1353 
Matched filter, 770-772 
Material derivative method, 493-494 
Mathematical shape theory, 1760 
MATLAB code, 75, 77 
Matrix-valued total variation, 1505 
Matsushita’s Divergences, 140-142 
Max formula, 278 
Maximal flow methods, 1477-1482 
Maximal violating pair strategy, 1448 
Maximum a posteriori (MAP) estimate, 751, 
2086 
Maximum a posteriori (MAP) estimation, 
1062, 1366 
Maximum entropy method, 1006 
Maximum likelihood (ML) method, 2083 
Maximum likelihood and Fisher information, 
1358-1359 
Maximum likelihood estimation, 306-309 
Maxwell’s equations, 661-666, 1172 
in Fourier domain, 1173-1174 
initial conditions of, 1174-1175 
Mean curvature motion (MCM), 1656-1657 
Measurement matrix, 214 
Measurement vector, 810 
Membership functions, 1896 
Mercer kernels, 1412 
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Mercer’s theorem, 1430-1431 
Merit function, 490 
Metric discretization 
diffusion distance, 1880-1882 
Dijkstra’s algorithm, 1874-1875 
eikonal equation, 1875-1876 
implicit surfaces and point clouds, 1880 
metrication errors and sampling theorem, 
1875 
parallel marching, 1878-1880 
parametric surfaces, 1878 
triangular meshes, 1876-1878 
Metric distortion, 1768 
Metric spaces, 1863 
diffusion distances, 1868 
diffusion geometry, 1866-1868 
Euclidean geometry, 1864 
isometries, 1863 
Riemannian geometry, 1864-1866 
topological spaces, 1863 
Metrication error, 1875 
Microlocal analysis, 871-883 
Microlocal analysis, X-ray CT, 884-887 
Microlocal regularity principle, 995 
Microwave breast screening, 475-477, 
523-525 
Middlebury database, 1962 
Migration imaging expectation, 
1316-1317 
Migration imaging SNR, 1318-1319 
Modeling error, 1383 
Modica-Mortola theorem, 1551 
Modulation operator, 1734 
Modulation transfer function, 1493-1494 
Moment conditions, 1142 
Moment generating function (MGF), 1058 
Momentum map, 1775 
Momentum representation, 1767 
Monge-Kantorovich formulation, 1989 
Monotone operator, 292 
Monotonicity method, 743 
Monotonicity properties 
Féjér monotonicity, 349 
function inequality, 353 
Gibbs smoothing, 359-362 
Shepp-Vardi EM algorithm, 350-352 
smoothed EM algorithm, 354-359 
Monte Carlo diffusion approach, 1044 
Morozov principle, 1007 
Morozov’s discrepancy principle, 34 
Morphological measures, 128-129 
Motion compensated filter, 1614 
Motion field, 1951 
MR image reconstruction, 199 
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Multi-frame blind deconvolution (MFBD), 52, 
78-81 

Multi-scale finite element approximation, 
1852-1853 

Multi-slice method, 956 
Multi-task-learning, 1423 

Multichannel total variation, 1504 
Multidimensional scaling (MDS), 1887 
Multiple anomalies detection, 543-544, 
559-560 

Multiplicative art algorithm (MART), 377-380 
Multiplicative noise removal, 195-196 
Multipolar fluids, 1828 

Multiresolution analysis, 1676, 1677, 1679, 
1681, 1685-1690 

Multiresolution support, 2077 

Multiscale vision model, 2075, 2076 
Multivariate Gaussian random variable, 817 
Mumford-Shah functionals, 498 

MUSIC, 631-635 

MUSIC-type algorithm, 543-544 
MUSIC-type imaging, single frequency, 
549-550 


N 
Nachman’s method, 754 
Nash embedding theorem, 1866 
Near-infrared (NIR) frequency, 1089 
Neighborhood filter/sigma filter, 1600, 1647 
NEMA NU 4-2008 phantom, 928 
Neumann function, 539 
Newton iterations, inverse obstacle scattering, 
672-675 
Newtonian fluid, 1827 
Newton’s method, 1805-1806 
Newton type methods, 447-458 
NL-means algorithm, 1609-1616 
Noise, 771 
Nonasymptotic bounds, 170-172 
Non-blind restoration, 1570 
Non-convex functionals, 1979-1981 
Non-convexity of S*4, 2014 
Non-convex model, 1420 
Nonconvex regularization, 172-178 
Nonlinear elasticity model, 1841 
Non-linear image registration 
distance functionals, 2010-2012 
ill-posedness and regularization, 2012 
image, 2008-2009 
mathematical setting, 2007-2008 
transformation, 2008-2010 
variational formulation, 2008 
Nonlinear Landweber regularization, 435-442 
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Nonlinear learning, 1410-1428 
Nonlinear maximization, 1370-1371 
Nonlinear problem frequency domain, 1265 
Nonlinear smoothing operator, 344 
Nonlinear statistical shape priors, 1918-1920 
Nonlocal Mumford-Shah regularizers, 
1584-1592 
Nonlocal total variation, 1507 
Non-negative solutions, for linear equations 
acceleration, 418-422 
general case, 418 
regularization, 418 
using prior bounds on, 420-423 
Non-physical scattering transform, 756 
Non-rigid volumetric objects, 1827-1834 
Non-smooth convex functionals, 1977-1979 
Nonsmooth regularization, 178-185 
Nonstationary inverse problems, 1063-1065 
Nonstationary Neumann-to-Dirichlet map, 
1207 
Non-stochastic EM algorithm (NSEM) 
continuous case, 391-393 
discrete case, 393 
Non-topological representations, 2118-2119 
Non-trapping condition, 1131 
Non-uniqueness set, 1127-1130 
Nonlinear problem frequency domain, 1265 
Normal cone mapping, 267 
Normal velocity, 496 
Notions of resolution, 1015 
Nuclear magnetic resonance (NMR) imaging, 
207 
Nuclear norm minimization, 227 
Nuisance parameters, 981 
Null hypothesis, 829 
Null space property, 216-217 
Nyquist-Shannon sampling theorem, 1952 


O 

Object classification, 2059 

Object reconstruction, 2092 

Obstacle scattering, 657-660, 667-671 

Ogden materials, 2015 

One-step late (OSL), 1374 

Online estimation methods, 1995 

Ontological scheme and seven-level hierarchy, 

2113-2116 

Optical coherence tomography (OCT), 1171 
forward operator of, 1181-1185 
frequency domain, 1171-1172 
full field, 1171 
measurements of, 1175-1177 
polarization-sensitive, 1172 
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standard, 1171 
time domain, 1171-1172 
Optical imaging 
adjoint field method, 1054-1055 
diffusion approximation, see Diffusion 
approximation (DA) 
error models, construction of, 1071-1072 
experiment and measurement parameters, 
1067-1070 
FEM meshes and discretization accuracy, 
1070-1072 
forward mapping, 1046-1047 
light propagation and probabilistic 
interpretation, 1056-1059 
linearization, 1052-1053 
MAP estimates, 1072-1074 
perturbation analysis, 1048-1049 
prior model, 1069-1071 
radiative transfer equation, 1037-1039 
Robin to Neumann map, 1046 
Schrodinger form, 1047-1048 
spectroscopic measurements, 1035-1036 
Optical tomography 
Bayesian framework, see Bayesian 
framework 
image reconstruction, 1059 
Optics, 957-962 
Optimal control formulation, 1811-1812 
Optimal current patterns, 713 
Optimal transport, 1988-1991 
Ordered subset EM algorithm (OSEM), 
371-374 
Order of approximation, 1686 
Orthogonality condition, 1142 
Orthonormal set, 23 
Overcomplete frame, 1724 


P 

Parabolic scaling, 1240 

Parallel beam geometry, 972 

Parallel beam transform, 869 

Parallel marching, 1878 

Parameter distribution, 482 

Parametric maximum flow algorithm, 1481 
Parametric shape representations, 1915-1920 
Parametrix approach, 1153-1155 
Parametrix-type reconstructions, 1158 
Parseval’s identity, 23 

Parseval’s theorem, 1867-1868 

Partial similarity, 1893-1896 

Partial symmetry, 1898 

Path-based shape spaces, 1822 

Path-based viscous dissipation, 1827-1831 
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Path optimization, 1807 
PDE models and local smoothing filters, 
1614-1618 
Perona-Malik model, 1618, 1625 
Persistent radar, 795 
Perturbation analysis 
Born approximation, 1048-1049 
Rytov approximation, 1049-1051 
Petroleum engineering, 477-479, 525-527 
Phase field approach, 1851-1852 
PhaseLift algorithm, 230 
Phase retrieval problem, 227, 229-230, 
987-988 
Photo-acoustic imaging, small absorbers 
bibliography and open questions, 585 
mathematical model, 580-581 
physical principles, 580 
reconstruction algorithms, 581-584 
Photoacoustic tomography (PAT), 1118-1163 
acoustic heterogeneities, 1103-1104 
data redundancies, 1102 
discrete imaging models, 1104—1106 
finite-dimensional object representations, 
1107-1108 
finite transducer bandwidth, 1096 
Fourier-Shell identity, 1093-1094 
frequency-dependent acoustic attenuation, 
1099-1100 
functional PAT, 1088-1089 
laser-based PAT, 1086-1088 
non-point-like transducers, 1097-1099 
RF-based PAT, 1087-1088 
speed-of-sound distribution, 1100-1102 
thermoacoustic effect and signal generation, 
1083-1086 
thermoacoustic tomography, see 
Photoacoustic tomography (PAT) 
universal backprojection algorithm, 
1092-1093 
Photometry, 2059 
Photon measurement density function (PMDF), 
1053 
Picard’s criterion, 29 
Picture distance measure, 826 
Picture function, 810 
Piecewise-constant mumford and Shah 
segmentation, 1558-1561 
Piecewise-smooth Mumford and Shah 
segmentation, 1561-1567 
Pixel, 813 
Pixelization, 965 
Planar detectors, 1123-1125 
Point detectors, 1120 
Point source method, 676 
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Point-spread function (PSF), 51, 784, 2094 

Pointwise perturbations, 548 

Poisson—Kirchhoff formula, 1121 

Poisson likelihood, 1374 

Poisson noise, 2070 

Poisson process, 1372 

Polar Cone Calculus, 286 

Polar format algorithm (PFA), 787 

Polarization tensor, 543 

Polarization tensor properties, 541 

Polyharmonic B-splines, 1694-1695, 
1701-1705 

Polynomial kernel, 1433 

Pontryagin maximum principle (PMP) 
theorem, 1773 

Positron emission tomography (PET), 20-21, 
51, 909, 2011 

maximum likelihood problem, 349 
Shepp—Vardi EM algorithm, 332-337 

Potential function (PF), 159, 172-173 

Potential method, 675 

Precision, 1362 

Pressure perturbations, 546 

Primal-dual active-set method, 
1519-1521 

Primal dual algorithm, 233-236 

Primal-dual approaches, 1475-1477 

Primal-dual hybrid gradient method, 
1518-1519 

Principal component analysis, 1821 

Prior, 817 

Prior density, 1348 

Probabilistic models, 1994-1995 

Probability density, 1375 

Probability density function (PDF), 309, 
817, 1058 

Probability distribution, 1346-1348 

Probe method, 637-639 

Projection approximation, 956 

Projection matrix, 816 

Projection method, 822 

Proximal mapping, 284 

Pseudodifferential operators, 875-880 

Pseudo-inverse operator, 1719 

Pshenichnyi—Rockafellar Conditions, 
287 

Puri-Vincze divergences, 140 

Pythagorean theorem, 7, 30 


Q 

Quadratic programming, 1527-1528 
Quantum mechanical models, 951 
Quantum mechanics, 970 
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Quasiphotons, 1231 
Quotient spaces, 1783-1784 


R 
Radar imaging, 232 
Rademacher’s theorem, 164 
Radial basis functions (RBF), 1654 
Radiative transfer equation (RTE), 1037-1039 
RAdio Detection And Ranging (Radar), 764 
SAR, see Synthetic aperture radar (SAR) 
imaging 
Radon line transform, 860-863 
Radon, Johann, 9 
Radon transform, 777, 810 
Radon’s inversion formula, 803 
Random media, imaging 
in cluttered media, 1303 
in forward model, 1283-1288 
in least squares inversion, 1292 
in long range scaling and Gaussian 
statistics, 1308-1309 
in normal operator, 1294-1295 
in passive arrays, 1295-1298 
in random model, 1303-1305 
in robustness to additive noise, 1299-1301 
in setup for imaging, 1313-1314 
in statistical moments, 1309-1313 
in time reversal process, 1292-1294 
in wave propagation, 1306-1307 
Random partial Fourier matrices, 222-223 
Range 
alignment, 788-791 
resolution, 785 
Range alignment, ISAR, 788-791 
Ray optics, 958-959 
Ray transform, 944 
Read-out noise, 965 
Reciprocity, 710 
Reciprocity gap function, 691 
Reconstruction algorithm, 810 
Reconstruction formula for OCT, 1185-1202 
for a dispersive layered medium with 
focused illumination, 1193-1196 
for a dispersive medium, 1189-1193 
for a non-dispersive medium, 1188 
for a non-dispersive medium with focused 
illumination, 1188-1189 
for an anisotropic medium, 1196-1202 
Reconstruction methods, 988-1014 
Reconstruction problem, 742-757 
Reconstruction problem, electron tomography, 
974-977 
Reconstruction procedure, 756 
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Redundant frame, 1724 
Reflectors, 1271-1273 
Region competition segmentation, 1634 
Region of interest (ROI) data, 888 
Regression, 1395 
Regression correction, neighborhood filter, 
1625-1628 
Regularization Concave on R,, 197-199 
Regularization parameters, 66-67 
Regularization scheme, 751 
Regularization theory, 32-34 
Regularized iterative non-linear methods, 
749-753 
Reinforcement learning, 1395 
Relative entropy, 142 
Relativistic corrections, 949 
Relaxation parameter, 819, 836 
Rellich’s lemma, 656, 663 
Reproducing kernel Hilbert spaces (RKHSs), 
1415-1419, 1428-1436, 1764-1765 
Resolution 
cross-range, 785-786 
range, 785 
Restoration of noisy signal, 183 
Restricted isometry property (RIP), 217-220 
Richardson-Lucy (RL) algorithm, 
2087-2089 
Ridge regression, 1397, 1408 
Riemannian geometry 
embedded manifolds, 1865 
geodesics, 1865 
n-dimensional manifold, 1864 
Riemannian metric tensor, 1865 
rigidity, 1866 
Riemannian manifolds, 1206, 1211-1214, 
1774, 1826-1827, 1865 
Riemannian metric, 1209, 1765 
Riemannian metric tensor, 1865 
Riemannian shape space, 1823 
Riemannian submersion, 1769-1770 
Riesz basis, 1678 
Riesz—Fredholm theory, 658 
Riesz potential, 787 
Riesz representation theorem, 24, 1763 
Right-inverse operator, 1719 
Rigid similarity, 1883-1885 
Rigid symmetry, 1897 
Robin boundary condition, 1043 
Robin to Dirichlet map, 1046 
Robust linear programming (RLP), 1420 
ROI tomography, 866-867 
Rollnick condition, 952 
Root-mean-square error (RMSE), 1657 
Rotating targets, 775 
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Row-action method, 821 
Rytov approximation, 1049-1051 


Ny 
Sampling and probe methods, 742 
Sampling theory, 1011 
Sandwich theorem, 278 
SAR imaging, 891-895 
Scalar curvature equation, 1221 
Scale-invariant heat kernel signatures 
(SI-HKS), 1901 
Scaled gradient projection (SGP), 2088 
Scaling function, 1677, 1679, 1683 
Scattered field model, 770 
Scattering operator, 948 
Scattering phase function, 1038 
Scattering relation, 1238-1240 
Schrédinger equation, 718-720 
Schrodinger form, 1047-1048 
Second dyadic decomposition, 
1240-1242 
Second generation starlet transform algorithm, 
2066-2068 
Second-order cone programming (SOCP), 
1529-1530 
Segmentation problem, 1540 
Selection rules, 1007 
Self-similarity and symmetry 
intrinsic symmetry, 1897 
partial symmetry, 1898 
repeating regular structure, 1899 
rigid symmetry, 1897 
spectral symmetry, 1897-1898 
Semi-blind restoration, 1570-1572 
Semi-smooth newton’s method, 1519 
Sensitivity functions, 1053-1054 
Separatrices, 2147-2148 
Sequential discrepancy principle, 100 
Series expansion method, 813 
Shannon entropy, 142 
Shape derivative, 495 
Shape discretization 
implicit surfaces, 1873 
parametric surfaces, 1873 
sampling, 1869-1870 
simplicial complexes, 1872-1873 
Shape distances, level sets, 1922-1923 
Shape distribution, 1884 
Shape DNA, 1893 
Shape evolution 
calculus of variations, 500-505 
Eulerian derivatives, 492-493 
geometric constraints, 497-499 
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Shape evolution (cont.) 
gradient direction, 501 
Heaviside function, 507-512 
least squares cost functionals, 500-501 
material derivative method, 493-494 
rough level set functions, 514 
and shape optimization, 516-518 
simple shapes and parameterized velocities, 
516 
smooth velocity fields, 515 
smoothed level set updates, 512-514 
of thin shapes, 504-505 
TM-waves, 503-504 
transformations and velocity flows, 
491-492 
velocity field, 502-503 
Shape identification problems, 592 
Shape sensitivity analysis, 492-493 
min-max principle, 506-507 
TM-waves, 506 
Short-time Fourier transform (STFT), 
1727-1730 
efficient Gabor expansion, 1746-1747 
modulation priority, 1747-1751 
translation priority, 1748, 1751 
visualization, 1747-1751 
Shot noise, 964-965 
Shrinkage estimators, 192 
Shrinking, 1448 
Shunt model, 704 
Signal-to-noise ratio (SNR), 771, 1071 
Silver—Muller finiteness conditions, 663 
Silver—Muller radiation conditions, 662 
Similarity filters, 1653-1655 
Simultaneous algebraic reconstruction 
technique (SART), 1004 
Simultaneous iterative reconstruction 
technique, 1004 
Simultaneous multiplicative algebraic 
reconstruction technique (SMART), 
377-380 
Single anomaly detection, 542-543, 557-558 
Single frequency 
backpropagation-type imaging, 550-551 
MUSIC-type imaging, 549-550 
Single photon emission computed tomography 
(SPECT), 909, 1372 
Singular sources method, 635-637 
Singular support and wavefront set, 871-874 
Singular value decomposition (SVD), 28, 
57-59 
Skeleton level, 2117 
Skin depth, 707 
Slater condition, 291 
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Small angle approximation, 955 
Small-angle case 
cross-range resolution, 785-786 
down-range resolution, 785 
Small-scene approximation, 772-773 
Smooth approximations, 1245 
Smoothness and support conditions, 1142 
SNARKOS9, 822 
Sobolev functions, 1542 
Sobolev metrics, 1794 
Sobolev spaces, 1212 
Soft field imaging, 710 
Soft margin classifier nonlinear learning, 
1402-1404, 1414 
Soft margin regression linear learning, 
1404-1407 
Soft margin regression nonlinear learning, 
1415 
Sommerfeld’s finiteness condition, 655 
Sonography, see Ultrasound imaging 
Source conditions, 434, 456 
Space-variant restoration, 1579-1582 
Sparse approximation, 210 
Sparse modeling, 2069 
Sparsity, 213-214 
Sparsity data model, 2061 
Sparsity promoting regularization, 1011-1013 
Spatio-temporal approach, 1973 
Spearman’s rank correlation coefficient, 151 
Spectral analysis, 912 
Spectral factorization, 59 
Spectral symmetry, 1897-1898 
Spectral theorem, 1429-1430 
Spherical Bessel function, 1142 
Spherical mean operator, 1121-1122 
Spiral CT, 805, 839 
Spline interpolation problem, 1764 
Spline reconstruction technique (SRT) 
PET, 918-922 
SPECT, 930-933 
Split Bregman iteration, 1523-1524 
Splitting methods, 1532-1534 
Spotlight SAR, 780 
Standard phase contrast model, 967-968, 979 
Star-galaxy separation, 2059 
Staring radar, 795 
Starlet reconstruction, 2064-2066 
Starlet transform, 2056 
Starlet wavelet transform, 2062 
State-based approach, 1826 
State-based elastic deformation, 1829-1833 
vs. path-based dissimilarity measure, 
1833-1834 
State estimation framework, 1064 
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State space representation, 1064 
Static relative permittivity, 475-477 
Statistical distance measures, 130-146 
Statistical hypothesis testing, 326 
Statistical methods 
additive noise model, 1354-1357 
counting noise, 1357 
hierarchical models, 1359-1362 
informative/noninformative priors, 1359 
maximum likelihood and Fisher 
information, 1358-1359 
maximum likelihood and maximum a 
posteriori estimation, 1363-1366 
Statistical significance, 827 
Steepest descent and minimal error method, 
446 
Stochastic EM algorithms, 313 
acceptable data, 396 
conventional formulation, 394 
E-step and M-step, 393-394 
incorrect proof, 395-396 
Stochastic filtering, 1995 
Stopping rules, 434-435 
Stratton—Chu formula, 662 
Streak line, 2142 
Structure determination problem, 939 
Structure tensor, 1969 
Subdifferential, 275-276 
subdifferential sum rule, 279 
Sublinear function, 274 
Sum-of-squared-difference (SSD), 2011 
Superiorization methodology, 822 
Superresolution, 1509-1511 
Surface impedance, lower bounds, 693-695 
SUSAN filter, 1602 
Synthesis operator, 1720 
Synthetic aperture radar (SAR) 
spotlight, 780 
Synthetic aperture radar (SAR) imaging, 
779-782, 856-858 
applications, radar imaging, 792-796 
historical background, 764-766 
mathematical analysis of methods survey, 
774-7186 
mathematical modeling, 766-773 
numerical methods, 787-791 
radar frequency bands, 765 
spotlight SAR, 780-782 
stripmap SAR, 781-782 
unmodeled motion, problems related to, 
792-793 
unmodeled scattering physics, problems 
related to, 792-796 
System matrix, 1108 
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T 
Tangent space, 1864 
Tangent vector, 1864 
Tangential divergence, 494-495 
Tangential vectors, 2104 
Tartaglia, Niccolo, 7 
Temporal coherence of silhouettes, 1932 
Tensors, 2105-2107 
TE-waves, 507 
Thermoacoustic tomography (TAT), 
1118-1163 
Tight frame, 1722 
Tikhonov regularization, 32-38, 96-98 
Tikhonov regularized solution, 1363 
Time-correlation single photon counting 
(TCSPC) systems, 1036 
Time-frequency lattice, 1731 
Time-frequency shifts (TFshifts), 1726 
Time line, 2142 
Time-reversal imaging, 552-554 
Time shift, 1726 
Tobacco mosaic virus (TMV), 1020 
Tomographic transforms properties, 859-871 
Tomography, 801-842 
Tomosynthesis, 53-55, 81-85 
Topological derivatives, 520-522 
Topological realization, 1872 
Topological skeletons, 2117-2118 
Topological space, 1863 
Topology, 2130-2132 
Total variation (TV), 1503-1507 
denoising problem, 266 
functionals, 498 
regularization, 161, 172, 1008-1011 
Trace-based PDE’s, 2152 
Traditional data model, 2057 
Transfer admittance, 710 
Transform method, 811 
Transform pairs construction, 911-912 
Transition function, 1864 
Transitive group action, 1784-1786 
Translation and scale invariance via alignment, 
1925-1926 
Translation operator, 1726 
Transmission eigenvalues, 695-696 
Transmission electron microscope (TEM), 807, 
940-943 
Transport equation, 1237-1238 
Transport theory, 1037 
Transportation map, 1989 
Trapping condition, 1131 
Triangle inequality, 1863 
Triangular meshes, 1872, 1876-1878 
Trilateral filters, 1652-1653 
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Turntable geometry, 775 
TV denoising problem, 1463-1468 
Two-dimensional case, 1623-1625 


U 
Ultrasound imaging, anomaly detection 
bibliography and open questions, 554-555 
frequency domain, asymptotic formulas, 
545-546 
numerical methods, 549-554 
physical principles, 545 
time domain, asymptotic formulas, 
547-548 
Itrasound imaging, by wave equation, 
1253-1275 
nbounded linear operators, 26 
nconditional basis, 1721 
nification, mathematical systems, 2134 
niform handling, different geometric 
primitives, 2135-2136 
niform uncertainty principle, 211 
nique continuation principle, 660 
niqueness of reconstruction, 
1135-1136 
niqueness set, 1127 
niversal backprojection algorithm, 
1092-1093 
Universal backprojection formula, 1149 


Geyer eG G 


eee 


Vv 

Vanderbilt database, 151 

Variable projection method, 70 
Variational image registration, 1987 
Variational image restoration, 1567-1569 
Variational inequalities, 106-107 
Variational methods, 1005-1013 
Variational principles, 291-292 
Variational regularization, 59-60, 98-104 
Vector and fiber bundles, 2110-2111 
Vector field visualization, 2141-2149 
Vector level set, 483 

Vector-valued case, 1628-1632 

Vertical bundle, 1780 
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Viscosity solution, 1876 

Viscous fluid shape space, 1843-1849 

Viscous moment tensor, 577 

Visibility condition, 1139-1140 

Visible (audible) singularities, 1136-1137 

Visualization, 805 

Voltage perturbations, asymptotic analysis, 
539-541 

Volterra iteration scheme, 1247 

Voronoi regions, 1870 


Ww 

Walking person, 1930-1931 

Wasserstein distances, 1885 

Wave equation, 766-767 

Wave equation model, 1119-1120 

Wavelet denoising, 2073 

Wave packets, 1210-1214 

Weak-scattering/single scattering 
approximation, 769 

Weighted back-projection (WBP), 813, 
990-994 

Weighted boundary measurements 
perturbations, 547 

Weil—Peterson metric, 1797 

Well-posed problem, 4 

Wentzel—Kramers—Brillouin (WKB) 
approximation, 954-955 

Weyl—Heisenberg frame, 1731 

White Gaussian noise Removal, 178, 192-194 


xX 
X-band, 765 
X-ray tomography (CT), 849-853 


Y 
Yosida approximation, 284 


Z 
Zero-boundary condition, 1041 
Zero level set, 480 


